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FOREWORD 


HumRRO  Work  Unit  UTILITY  was  initiated  in  January  1967  as  part  of  Project 
100,000  to  provide  information  about  the  Army  performance  and  characteristics  of 
mar^nal  personnel,  men  whose  scores  on  the  Armed  Forces  Qualification  Tests  were  in 
the  low  levels.  Its  objectives  were  (a)  to  find  out  how  men  in  Mental  Category  IV 
compared  with  men  in  other  mental  categories  in  the  performance  of  selected  Army  jobs, 
and  (b)  to  identify  different  factors  associated  with  satisfactory  performance  in  different 
mental  category  groupings— specifically,  to  explore  the  relationships  among  a  man’s 
background,  personal  characteristics.  Army  experiences,  and  job  performance. 

The  UTILITY  research  has  been  conducted  by  HumRRO  Division  No.  3  at  the 
Presidio  of  Monterey,  California,  with  Dr,  Howard  McFann  as  Director.  Dr.  Robert 
Vineberg  was  the  Work  Unit  Leader.  Members  of  the  research  team  at  various  times  have 
included  Dr.  Elaine  N.  Taylor,  Dr.  John  S.  Caylor,  Miss  Annette  K,  Mahikoa,  Dr.  S. 
James  Goffard,  Dr.  Thomas  G.  Sticht,  Dr.  Joseph  S.  Ward,  Dr.  Herbert  G.  Gerjuoy,  Mr. 
Donald  F.  Polden,  and  Mr.  Leon  E.  Guyton. 

Military  support  for  the  study  was  provided  by  the  U.S.  Army  Training  Center 
Human  Research  Unit.  Military  Chief  of  the  Unit  at  the  beginning  of  the  research  was 
LTC  David  S.  Marshall;  during  the  period  of  data  collection  and  early  analysis,  LTC 
Robert  J.  Emswiler  was  Unit  Chief;  the  present  Chief  is  COL  Ullrich  Hermann. 

Enlisted  men  assigned  to  the  project  during  the  data  analysis  and  report  preparation 
phases  included  SGT  Gerald  G.  Lynch,  SP5  William  Yanda,  and  SP4  Gregory  Herr. 

The  extensive  findings  from  this  research  are  being  reported  in  a  series  of  publica¬ 
tions.  This,  the  fourth  report,  examines  the  determinants  of  job  behavior  and  describes 
the  relationships  among  the  three  performance  criteria  used  in  the  study:  job  sample 
tests,  job  knowledge  tests,  and  supervisor  ratings.  The  first  report  in  the  series  described 
the  rationale,  research  design,  and  general  chronology  of  research  events.  The  second 
report  described  the  data  collection  instruments  used  in  the  study  and  their  development 
and  administration.  The  third  report  presented  the  bulk  of  the  major  study  findings, 
including  comparisons  of  the  performance  of  men  in  different  mental  categories  with 
different  amounts  of  job  experience;  comparisons  of  the  performance  of  speciad  sub¬ 
groups  (Blacks  and  Whites,  Inductees  and  Enlistees,  and  Men  with  Formal  and  On-the- 
Job  Training);  an  analysis  and  definition  of  acceptable  performance;  a  procedure  for  using 
job  knowledge  tests  to  screen  ineffective  performers;  and  an  analysis  of  the  relationship 
between  personal  characteristics  and  job  performance. 

HumRRO  research  for  the  Department  of  the  Army  is  conducted  under  Contract 
DAHC  19-70-C-0012.  Training,  Motivation,  Leadership  Research  is  conducted  under 
Army  Project  2Q062107A712. 


Meredith  P.  Crawford 
President 

Human  Resources  Research  Organization 
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SUMMARY  AND  CONCLUSIONS 


INTRODUCTION 

Work  Unit  UTILITY  was  concerned  with  a  comparison  of  the  job  proficiency  of 
men  in  Mental  Category  IV  and  other  mental  categories  in  four  different  Army  jobs: 
Armor  Crewman,  General  Vehicle  Repairman,  Unit  and  Organizational  Supply  Specialist, 
and  Cook.* 


PROBLEM 

A  fundamental  problem  in  the  assessment  of  job  proficiency  is  the  selection  of 
appropriate  and  satisfactory  criterion  instruments.  The  type  of  measure  to  be  used 
depends  upon  a  variety  of  considerations,  including  the  purpose  and  situation  in  which 
assessment  is  being  undertaken,  the  feasibility  of  employing  a  specific  type  of  measure  in 
a  particular  operational  or  research  setting,  and  the  costs  in  time,  personnel,  and  money 
that  are  required.  Because  practical  constraints  inevitably  can  be  expected  to  promote  the 
use  of  criterion  instruments  that  are  simplest  to  administer,  it  becomes  essential  to 
examine  both  the  qualities  needed  in  criterion  instruments  and  the  relationships  between 
different  types  of  criterion  instruments  to  determine  where  substitutions  of  one  for 
another  can  reasonably  be  considered.  This  report  is  designed  to  provide  the  reader  with 
an  analysis  and  a  general  review  of  the  factors  that  govern  the  proper  selection  of 
criterion  measures  for  different  situations  and  purposes. 

In  UTILITY  three  different  kinds  of  criterion  instruments  were  used:  job  sample 
tests,  job  knowledge  tests,  and  supervisor  ratings.  In  general,  these  three  classes  of 
criterion  measures  differ  conceptually  in  their  approach  to  the  assessment  of  job 
proficiency,  and,  as  a  consequence,  can  focus  on  quite  different  elements  of  job  behavior. 
Job  sample  tests  are  based  upon  samples  of  the  performances  required  in  a  job.  Job 
knowledge  tests  are  based  upon  an  analysis,  extraction,  and  sampling  of  the  knowledge 
required  in  job  performance.  Supervisor  ratings  are  based  upon  verbal  descriptions  of  job 
behavior  or  determinants  of  job  behavior.  The  data  collected  in  UTILITY  provide  a 
means  for  an  empirical  examination  of  the  relationship  among  these  three  different 
varieties  of  criterion  performance  in  four  different  jobs. 


APPROACH 

In  the  selection  of  a  criterion  instrument,  a  general  understanding  of  the  primary 
determinants  of  job  behavior  is  required.  These  are: 

Job  Knowledge— Information  about  material  used  on  the  job  and  about  acts, 
procedures,  and  principles  needed  to  mediate  job  performance. 

Job  Skill— Performances  not  mediated  completely  by  knowledge  or  conscious 
verbal  processes  (self-instruction)  and  not  readily  acquired  solely  through  vebal  instruc¬ 
tion.  Skills  include  perceptual,  motor,  cognitive,  and  social  skills. 


*  Information  about  the  Medical  Specialist,  a  fifth  MOS  included  in  the  study,  is  not  contained  in 
this  report.  Job  specialization  in  this  MOS  was  found  to  be  considerable.  Sampling  procedures  and 
analysis  of  these  data  differed  from  those  followed  for  the  other  four  jobs,  and  for  this  reason 
information  on  this  MOS  has  not  been  included  here. 


Motivation  and  Personality  Characteristics— Attitudes,  values,  and  general  styles 
of  behavior  that  condition  the  manner  in  which  knowledge  and  skill  are  employed  in  job 
performance. 

Selection  of  an  appropriate  criterion  instrument  depends  upon  the  relative  con¬ 
tribution  of  each  of  these  factors  to  performance  in  a  specific  job  situation.  Selection  of 
a  criterion  instrument  also  depends  upon  the  purpose  for  which  measurement  is  being 
undertaken.  The  major  purposes  of  job  proficiency  measurement  can  be  summarized  as 
follows: 

To  provide  estimates  of  the  effectiveness  of  an  individuaTs  job  performance. 

To  diagnose  deficiencies  during  training  or  on  the  job. 

To  evaluate  the  effectiveness  of  a  training  program. 

To  provide  estimates  of  group  performance  in  research  studies. 

The  rationale  that  underlay  the  selection  of  the  three  types  of  criterion  instruments  (job 
sample,  job  knowledge,  and  supervisor  ratings)  used  in  UTILITY  is  presented  in  the 
report. 

As  observed  earlier,  practical  constraints  often  dictate  that  relatively  simple  and 
inexpensive  measures  be  used  to  assess  job  proficiency,  sometimes  at  the  sacrifice  of  more 
valid  and  appropriate  information.  The  report  explores  the  interchangeability  of  job 
knowledge  and  job  sample  tests,  and  the  considerations  that  govern  the  suitability  of  such 
substitution.  An  analysis  was  made  of  the  skill  and  knowledge  requirements  of  the  job 
sample  tests  that  are  considered  to  exemplify  the  demands  of  the  jobs  themselves.  The 
analysis  suggests  that  well-constructed  knowledge  tests-Hhat  is,  tests  that  contain  only 
information  required  in  actual  job  performance— could  be  substituted  for  job  sample  tests, 
at  least  in  the  four  jobs  studied. 

The  reliabilities  and  interrelationships  of  job  sample  tests,  job  knowledge  tests,  and 
supervisor  ratings  were  then  determined.  These  data  support  the  analyses  and  conclusions 
already  reached. 

The  final  step  in  the  review  was  a  study  of  the  relationships  between  job  sample 
tests  and  job  knowledge  tests  for  selected  subsamples:  low  vs.  high  reading  ability  groups, 
Mental  Category  IV  vs.  Non-IV  groups,  low  vs.  high  verbal  aptitude  groups,  low  vs.  high 
educational  level  groups,  and  black  vs.  white  groups. 


RESULTS  AND  CONCLUSIONS 

(1)  Correlations  between  job  sample  test  scores  and  job  knowledge  test  scores  in  the 
four  jobs  ranged  from  .58  to  .72.  Job  knowledge  test  reliability  coefficients  ranged 
from  .76  to  .92.  Estimates  of  the  lower  lunit  of  job  sample  test  reliabilities  ranged 
from  .35  to  .55. 

(2)  Correlations  between  job  sample  test  scores  and  supervisor  ratings  ranged  from 
.20  to  .28.  Correlations  between  job  knowledge  test  scores  and  supervisor  ratings  ranged 
from  .23  to  .35. 

(3)  Analysis  of  the  job  sample  tests  used  to  measure  performance  for  the  four  jobs 
in  this  study— Armor  Crewman,  Repairman,  Unit  and  Organizational  Supply  Specialist, 
and  Cook— showed  that  the  skill  requirements  in  these  jobs  are  minimal.  Therefore,  job 
knowledge  tests  could  appropriately  be  substituted  for  more  direct  measures  of  perform¬ 
ance  such  as  job  sample  tests  in  assessing  proficiency  in  these  jobs. 

(4)  The  selection  of  appropriate  criterion  measures  for  assessing  job  proficiency 
requires  that  the  skill  and  knowledge  components  of  a  job  be  identified.  Where  job 


VI 


performance  is  mediated  almost  solely  by  knowledge  per  se,  job  knowledge  tests  should 
be  used  to  assess  the  information  a  man  has  about  his  job.  Where  job  performance  relies 
almost  solely  on  skill,  job  sample  tests  or  some  other  variety  of  performeince  measure  is 
essential.  Where  job  performance  depends  upon  a  mix  of  knowledge  and  skill,  both 
knowledge  tests  and  performance  tests  are  indicated.  An  instrument  that  has  been 
designed  to  assess  job  motivation  and  other  personal  qualities,  such  as  rating  scales,  is 
always  essential  if  a  complete  estimate  of  job  performance  is  desired. 
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Performance  in  Four  Army  Jobs  by 
Men  at  Different  Aptitude  (AFQT)  Levels: 
4.  Relationships  Between 
Performance  Criteria 


INTRODUCTION 


BACKGROUND 

The  most  fundamental  and  difficult  problem  in  any  attempt  to  measure  job 
proficiency  is  the  selection  of  appropriate  and  satisfactory  criteria.  The  choice  of  a 
particular  measure  or  criterion  of  proficiency  depends  upon  a  variety  of  considerations, 
including  the  purpose  or  purposes  for  which  assessment  is  being  undertaken,  its  feasibility 
for  use  in  a  particular  operational  or  research  setting,  and  its  cost.  An  important  factor 
determining  the  suitability  of  a  criterion  is  the  extent  to  which  it  measures  aspects  or 
determinants  of  behavior  that  differentiate  among  performers  in  a  particular  job  with 
regard  to  factors  that  are  critical  determinants  of  performance  in  that  job. 

This  report  examines  the  suitability  of  different  criteria  in  relation  to  the  specific 
behavioral  demands  of  a  job.  In  particular,  it  focuses  on  the  usefulness  and  relationships 
among  three  types  of  criterion  instruments:  job  sample  tests,  job  knowledge  tests,  and 
supervisor  ratings.  Each  of  these  instruments  has  been  used  to  assess  men  working  in  the 
four  Army  jobs — Armor  Crewman,  General  Vehicle  Repairman,  Unit  and  Organizational 
Supply  Specialist,  and  Cook— that  were  included  in  Work  Unit  UTILITY,  a  comparative 
study  of  the  performance  of  men  in  different  mental  ability  groups. 

The  primary  objectives  of  the  overall  UTILITY  research  program  were  to  find  out 
how  men  in  Category  IV*  and  other  mental  categories  compare  in  the  performance  of 
selected  Army  jobs,  and  to  identify  different  factors  associated  with  satisfactory  perform¬ 
ance  in  different  mental  category  groupings.  The  purpose  and  design  of  Work  Unit 
UTILITY  have  been  described  in  an  earlier  report  in  this  series  (1). 

Other  reports  have  described  the  data  collection  instruments  used  in  the  study, 
including  the  criterion  measures  (2),  and  job  performance  as  it  varies  both  in  accordance 
with  the  main  study  variables— AFQT  level  and  amount  of  job  experience— and  in 
different  ethnic  groups  (3). 

In  general,  the  three  different  kinds  of  criterion  instruments  used  in  the  UTILITY 
research— job  sample  tests,  job  knowledge  tests,  and  supervisor  ratings— differ  conceptually 
in  their  approach  to  the  assessment  of  job  proficiency.  As  a  consequence,  they  can  focus 
on  quite  different  elements  of  job  behavior: 

(1)  Job  sample  tests  are  based  upon  samples  of  performance  required  in  a  job. 
Where  performance  is  mediated  almost  entirely  by  knowledge,  a  valid  job  sample  test  will 
necessarily  measure  knowledge.  Where  performance  is  mediated  almost  entirely  by  skill,  a 
valid  job  sample  test  measures  largely  skill.  Where  job  performance  is  mediated  by  both 
knowledge  and  skill,  valid  job  sample  tests  measure  both  knowledge  and  skill. 

(2)  Job  knowledge  tests  are  based  upon  an  analysis,  extraction,  and  sampling 
of  the  knowledge  required  in  job  performance  and  are  intended  solely  to  measure 
knowledge.  They  cannot  be  used  to  measure  skill. 

(3)  Supervisor  ratings  are  based  upon  verbal  descriptions  of  job  behavior  or 
determinants  of  job  behavior.  Although  supervisor  ratings  can  be  applied  to  the 


*This  includes  men  in  Mental  Categories  IV  and  V  (i.e.,  men  who  received  a  percentile  score  of 
0-30  on  the  Armed  Forces  Qualification  Test,  (AFQT).  Men  in  these  categories  are  considered  marginal 
and  their  acceptance  into  the  military  depends  upon  the  minimum  standard  prevailing  at  the  time. 
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measurement  of  many  different  aspects  of  job  behavior,  they  generally  provide  more 
global  estimates  of  performance,  such  as  attitude  toward  the  job,  and  general  personal 
adaptability  and  reliability. 

In  the  UTILITY  research,  job  sample  tests  were  viewed  as  providing  the  closest 
approximation  to  the  ultimate  criterion  of  actual  job  performance.  Job  knowledge  tests 
were  used  because  they  provide  a  very  efficient  means  of  sampling  from  the  entire  range 
of  knowledge  required  in  job  performance.  Supervisor  ratings  were  used  because  they 
provided  the  only  feasible  means  for  assessing  motivation  and  other  personal 
characteristics  that  a  job  incumbent  had  demonstrated  in  his  job.  Also,  these  three 
different  types  of  criterion  measures  were  included  in  the  research  to  permit  a  study  of 
the  interrelationships  among  the  measures. 

It  is  generally  recognized  that  different  criteria  vary  in  their  suitability  for  measuring 
different  components  of  performance,  that  criteria  vary  in  their  degree  of  relevance  for 
assessing  proficiency  in  given  jobs,  and  that  one  variety  of  criterion  performance  is  not 
necessarily  closely  related  to  another.  Such  understanding  is,  however,  most  often 
honored  in  the  breach.  For  example,  in  the  Army  even  in  a  job-image  interpretation- 
requiring  extensive  perceptual  skill,  a  man’s  proficiency  is  assessed  on  the  basis  of  a 
composite  of  his  scores  on  verbal  or  paper-and-pencil  tests  of  fact  and  principle  and  a 
supervisor’s  ratings  of  his  personal  characteristics.^ 

Reasons  for  using  tests  of  information  or  knowledge  or  summary  ratings  of  job  and 
general  effectiveness  in  situations  where  tests  of  performance  are  far  more  appropriate 
are,  of  course,  not  difficult  to  find.  Job  sample  tests  are  not  convenient;  they  are 
expensive  in  money,  in  time,  and  in  the  professional  skill  required  in  test  development 
and  supervision  of  the  testing  program.  The  use  of  job  sample  tests  in  operational 
assessment  programs,  therefore,  is  often  not  practicable. 

Because  practical  constraints  inevitably  can  be  expected  to  promote  the  use  of  the 
criterion  instruments  that  are  simplest  to  administer,  it  becomes  essential  to  examine 
both  the  qualities  needed  in  criterion  instruments  and  the  relationship  among  different 
types  of  criterion  instruments  in  order  to  determine  where  it  is  reasonable  to  consider 
substitutions  of  one  for  another. 


OBJECTIVES  OF  THE  STUDY 

The  data  collected  in  Work  Unit  UTILITY  provide  a  means  for  examining  the 
relationship  among  different  varieties  of  criterion  performance  in  four  fairly  different 
types  of  jobs.  Armor  Crewman  job  duties,  dictated  in  a  large  degree  by  equipment 
demands,  are  highly  procedural;  the  tasks  may  be  considered  machine-ascendant,  in  that 
job  behaviors  principally  involve  sequences  of  interaction  between  man  and  machine,  in 
which  a  man’s  activities  are  largely  determined  by  characteristics  of  the  equipment. 
General  Vehicle  Repairman  is  a  mechanical  maintenance  job  requiring  diagnostic  and 
interpretive  behaviors.  Supply  Specialist  is  a  clerical  job  calling  primarily  for  the  analysis 
and  recording  of  information.  The  Cook’s  job  typically  requires  reading  and  following 
procedures  specified  in  a  cookbook;  recognition  of  standards  and  precision  in  meeting 
them  are  primary  characteristics  of  the  Cook’s  job. 

The  analysis  of  criterion  test  selection  factors  includes  the  following: 

(1)  An  analysis  of  the  primary  determinants  of  job  behavior. 


^Department  of  the  Army,  “Proficiency  Pay  for  Army  Enlisted  Personnel,”  Chapter  6,  Enlisted 
Personnel  Management  System^  (AR)  600-200,  Washington,  March  1966. 
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(2)  A  classification  of  proficiency  measures,  specification  of  the  major  purposes 
of  proficiency  measurement,  and  discussion  of  the  rationale  underlying  the  selection  of 
the  three  instruments  used  in  UTILITY. 

(3)  A  discussion  of  the  interchangeability  of  job  sample  and  job  knowledge 
tests  and  an  analysis  of  the  skill  and  knowledge  requirements  in  the  four  jobs  studied  in 
UTILITY. 

(4)  A  presentation  of  the  reliabilities  and  interrelationships  of  the  three 
criterion  instruments. 

(5)  A  presentation  of  the  relationships  between  job  sample  tests  and  job 
knowledge  tests  for  selected  subsamples  of  the  study. 


COMPONENTS  OF  JOB  PERFORMANCE 

A  first  step  in  relating  the  suitability  of  different  types  of  criterion  measures  to  the 
characteristics  of  jobs  is  the  identification  of  relevant  components  of  performance  that 
can  be  used  to  classify  jobs.  Several  investigators  have  undertaken  the  classification  of 
tasks,  training  objectives  (tasks  to  be  learned),  and  jobs  (4,  1_,  8,  9,  1^,  1^).  Some 

of  these  efforts,  such  as  that  of  Fine  (5),  are  quite  comprehensive  and  detailed.  For 
present  purposes,  a  relatively  simple  set  of  job  components  or  determinants  has  been 
adopted: 

(1)  Job  Knowledge.  Information  about  material  used  on  the  job  and  about 
acts,  procedures,  and  principles  needed  to  mediate  job  performance. 

(2)  Job  Skill.  A  rather  loose  grouping  of  different  varieties  of  behavior  not 
mediated  completely  by  knowledge  or  conscious  verbal  processes  (self-instruction)  and 
not  readily  acquired  solely  through  verbal  instruction.  Thus,  job  skill  is  usually,  thought 
of  as  being  able  to  do  something  rather  than  knowing  what  to  do.  Included  are 
perceptual  skills,  motor  skills,  cognitive  skills,  and  social  skills. 

(3)  Motivation  and  Personality  Characteristics.  Attitudes,  values,  and  general 
styles  of  behavior  that  condition  the  manner  in  which  knowledge  and  skill  are  employed 
in  job  performance.^ 


JOB  KNOWLEDGE 

Information  about  a  job  generally  can  be  considered  the  single  most  important 
mediator  of  job  behavior.  A  knowledge  of  specific  acts  and  procedures  guides  the  worker 
through  the  performance  of  most  work  activities.  Knowledge  in  the  form  of  facts, 
principles,  and  relationships  (sometimes  in  conjunction  with  the  application  of  cognitive 
skills)  enables  an  incumbent  to  respond  to  contingency  situations  and  to  accommodate  to 
situations  where  procedures  must  be  modified,  or  where  novel  forms  of  response  are 


^  Omitted  from  the  set  of  performance  components  are  such  determinants  of  behavior  as  aptitude 
and  physical  ability.  Aptitudes  are  general  abilities  that  condition  the  acquisition  of  knowledge  and  the 
development  of  skills.  This  analysis  focuses  on  knowledge  and  skill  requirements  in  performance  rather 
than  upon  factors  affecting  the  development  of  these  capabilities  in  the  performer.  Physical  abilities  are 
characteristics  of  physical  structure  and  function  that  permit  job  acts  to  be  carried  out.  While  clearly  an 
important  determinant  of  performance,  physical  ability  has  been  omitted  from  the  set  of  behavior 
determinants  since  most  men  generally  possess  the  physical  abilities  required  to  perform  the  majority  of 
jobs.  Where  special  needs  exist  (e.g.,  tea  taster,  stevedore,  athlete)  such  factors  would  obviously  have  to 
be  represented  among  the  criteria  for  assessing  proficiency. 
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required.  Job  knowledge  may  exist  as  information  that  the  incumbent  brings  to  the  job, 
or  it  may  be  provided  in  the  form  of  job  instructions  or  job  aids. 


JOB  SKILL 

Job  skill  refers  to  a  heterogeneous  grouping  of  abilities.  These  abilities  are  inferred 
on  the  basis  of  behaviors  that  are  not  completely  mediated  by  knowledge  or  verbalization 
of  a  to-be-performed  act.  The  acquisition  of  skill““being  able  to  do  something  rather  than 
knowing  what  to  do-mnlike  the  acquisition  of  knowledge,  requires  practice  or  rehearsal. 
Indeed,  the  need  for  practice  in  its  development  can  be  considered  the  major  definitional 
characteristic  of  a  skill.  This  characteristic  is  particularly  important,  because  it  provides  a 
basis  for  deciding  whether  or  not  skill  is  being  manifested  in  any  given  behavior,  a 
decision  that  is  critical  in  the  selection  of  appropriate  criterion  instrumentation.  Thus, 
just  as  the  description  or  specification  of  skilled  behavior  is  insufficient  to  mediate 
performance  of  that  behavior,  so  expression  of  knowledge  of  skilled  behavior  by  an 
incumbent  (e.g.,  job  knowledge  test)  is  insufficient  reason  for  concluding  that  the  skilled 
behavior  can  be  demonstrated. 

A  related  and  common,  although  probably  not  essential,  characteristic  of  skill  is  that 
skilled  behavior  typically  is  not  completely  specifiable  by  verbal  means  (although  the 
ends  of  such  behavior  are).  For  example,  a  person  cannot  be  told  by  verbal  instruction 
how  to  recognize  subtle  shadings  in  pitch  in  auditory  tones  or  in  a  spoken  language  or 
how  to  swing  a  baseball  bat.  These  behaviors  can  be  demonstrated,  but  cannot  be 
completely  reduced  to  verbal  description,  primarily  because  the  mechanisms  underlying 
them  are  internal  and  nonverbal  (auditory  processes  in  the  case  of  discriminating  tones, 
kinesthetic  processes  in  swinging  a  bat),  and  language  generally  does  not  provide 
commonly  agreed  upon  verbal  labels  for  describing  such  processes. 

Four  main  classes  of  job  skill  can  be  identified:  perceptual  skills,  motor  skills, 
cognitive  skills,  and  social  skills.  While  for  some  purposes  it  may  be  useful  to  distinguish 
additional  varieties  of  skill,  such  as  artistic  skills,  these  four  classes  seem  to  cover  the 
critical  varieties  of  skill  in  most  jobs. 

Perceptual  Skills 

Perceptual  skills  involve  the  ability  to  discriminate,  interpret,  and  organize  sensory 
information  (e.g.,  recognizing  and  interpreting  camouflaged  targets  in  aerial  photographs, 
or  receiving  International  Morse  Code).  The  development  of  perceptual  skills  such  as 
those  involved  in  photo  interpretation  requires  practice  during  which  a  person  learns  to 
discriminate  varieties  and  combinations  of  subtle  visual  cues.  Also,  with  practice,  the 
identification  process  undergoes  a  shortcutting  or  dropping  out  of  verbal  self-instruction 
or  cue-naming  that  initially  aids  in  mediating  performance. 

This  process  can  be  more  clearly  observed  in  the  development  of  skill  in  receiving 
code.  Here  the  learner  first  verbalizes  the  name  of  each  letter  to  himself  as  he  hears  its 
signal.  He  then  records  the  letter.  With  practice,  the  naming  step  drops  out  and  the 
auditory  signal  becomes  a  cue  for  the  direct  v^nriting  of  letters. 

The  learning  of  Morse  Code  also  demonstrates  another  common  feature  of  skill 
acquisition.  With  practice,  the  learner  organizes  the  incoming  signals  into  larger  and  larger 
groupings  or  hierarchies.  Initially  he  attends  to  signals  which  denote  letters;  subsequently, 
he  attends  to  and  “hears”  a  longer  sequence  of  signals  that  signify  words  and  ultimately 
phrases. 
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Motor  Skills 


Motor  skills  (e.g.,  tracking  and  leading  a  moving  target  with  a  rifle,  or  shifting  from 
neutral  in  an  automobile  in  which  the  simultaneous  letting  out  of  the  clutch  and 
depression  of  the  accelerator  are  coordinated)  involve  the  ability  to  make  appropriate 
physical  responses.  During  the  development  of  motor  skills,  a  variety  of  typical  processes 
may  occur.  The  coordination  and  timing  of  several  simultaneous  or  sequential  movements 
may  be  required,  as  in  shifting  gears  in  an  automobile.  The  connection  and  smoothing  of 
responses  into  a  rhythmic  sequence  may  be  needed,  as  in  dancing.  The  combination  and 
linking  of  a  series  of  initially  discrete  responses  and  adjustments  may  be  necessary,  as  in 
combining  into  a  single  act  the  elements  involved  in  returning  a  tennis  ball.  As  is  the  case 
for  many  perceptual  skills,  the  development  of  motor  skill  often  requires  learning  to 
make  rapid  and  automatic  responses  (without  self-instruction  on  the  part  of  the  per¬ 
former)  or  precise  or  delicate  manual  movements.’ 

Cognitive  Skills 

Cognitive  skills  involve  those  covert  mental  processes  associated  with  the  develop¬ 
ment  of  concepts  or  rules,  understanding  the  implications  of  relationships,  and  applying 
rules  and  principles  in  making  decisions  and  solving  problems.  In  the  present  analysis, 
behaviors  have  been  designated  as  skilled  when  knowledge  or  a  description  of  the 
behavior  in  question  is  not  sufficient  to  ensure  its  performance  and  where  practice  is 
necessary  to  produce  certain  changes  and  refinements  in  initial  attempts  to  perform. 

When  cognitive  behavior  is  examined,  one  finds  the  acts  or  processes  themselves 
almost  impossible  to  describe.  Such  behavior  is  only  poorly  understood.  The  lack  of  an 
external  referent  to  its  functioning  and  the  absence  of  language  for  describing  its 
underlying  processes  make  it  particularly  resistant  to  description.  It  is  difficult  to  describe 
how  hypotheses  are  generated  other  than  by  way  of  illustration.  A  person  can  be  told 
how  to  go  about  identifying  his  opponent’s  likely  moves  in  a  chess  game  or  how  to  select 
electronic  components  for  testing  in  a  troubleshooting  task,  but  he  cannot  be  told  how 
to  develop  hypotheses  in  the  abstract,  mainly  because  it  is  impossible  to  specify  the 
dimensions  of  anything  but  a  concrete  situation. 

Because  cognitive  processes  cannot  be  described  adequately  or  reduced  to  statements 
of  how  cognitive  acts  are  to  be  carried  out,  the  question  of  whether  the  performance  of 
acts  of  cognition  requires  something  more  than  knowledge  of  those  acts  is  largely 
unanswerable.  We  are  unable  to  construct  a  knowledge  test  consisting  solely  of  questions 
about  how  cognitive  acts  are  accomplished,  therefore,  possession  of  knowledge  of  the 
form  of  a  cognitive  act  cannot  be  separated  from  the  performance  of  the  act  itself. 

Because  a  knowledge  test  cannot  be  built,  it  is  not  possible  to  establish  the 
relationship  between  cognitive  knowledge  and  the  actual  performance  of  cognitive  acts.  It 
is  appropriate,  therefore,  to  treat  cognitive  behavior  as  a  skill  not  because  cognitive 
performance  necessarily  requires  something  more  than  knowledge  of  how  cognitive  acts 
are  accomplished,  but  because  knowledge  cannot  be  demonstrated  apart  from  perform¬ 
ance.  Because  knowledge  of  cognitive  acts  cannot  be  independently  described  and 
measured,  the  assessment  of  cognitive  skill  will  necessarily  involve  the  direct  performance 
of  cognitive  acts. 

The  role  of  practice  or  training  in  the  acquisition  of  cognitive  skill  is  also  poorly 
understood.  Whether  or  not  practice  is  always  essential  in  the  development  of  such  skill  is 
not  clear.  Nevertheless,  cognitive  skills  do  improve  with  practice,  and  typically  the  only 

’  As  indicated  earlier,  skill  development  presupposes  certain  aptitudes  and  physical  abilities; 
practice  in  and  of  itself  is  not  necessarily  sufficient.  For  example,  women  seem  more  likely  to  possess 
the  manual  dexterity  required  for  the  soldering  of  miniaturized  equipment;  some  adults  appear  unable  to 
recognize  subtle  infleetions  in  a  foreign  language  regardless  of  how  much  training  they  receive. 
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means  available  for  effecting  this  development  is  to  expose  persons  to  situations  where 
cognition  is  required.  The  means  by  which  a  person  learns  to  process  information,  see 
relationships,  and  make  inferences  and  deductions  in  order  to  make  decisions  and  solve 
problems  are  not  known.  Thus,  training  to  develop  cognitive  skill  is  inevitably  reduced  to 
a  series  of  demonstrations  whereby  a  learner  is  exposed  to  examples  of  problems  and 
given  practice  in  dealing  with  them.  In  this  sense,  the  development  of  cognitive  skill  is 
not  dissimilar  from  the  development  of  other  varieties  of  skill. 

Social  Skills 

Social  skills  involve  the  ability  to  work  successfully  with  individuals  and  groups  of 
people.  In  the  present  analysis,  social  skills  refer  to  any  variety  of  interpersonal  skill  and 
behavior  that  is  necessary  to  accomplish  a  specific  job  objective.  While  these  skills  may 
come  into  play  in  any  work  situation  involving  two  or  more  persons,  the  skills  associated 
with  effective  supervisory  performance  and  effective  group  performance  are  of  particular 
importance. 

Social  skills  have  been  grouped  with  the  other  varieties  of  skill  because  precepts  or 
general  principles  (knowledge)  for  effective  social  interaction  do  not  necessarily  lead  to  a 
recommended  action.  A  complete  set  of  principles  for  effective  social  interaction  is  not 
available,  and  existing  principles  are  nonspecific.  Rules  for  identifying  situations  in  which 
they  apply  and  the  behaviors  they  call  for  are  not  clearly  defined.  Thus,  in  responding  to 
an  exhortation  such  as  “be  firm  yet  sensitive  to  the  feelings  of  others,”  it  is  not  certain 
what  weight  is  to  be  given  to  these  somewhat  contradictory  aspects  of  behavior.  The 
specific  behavior  called  for  in  accommodating  to  both  perceived  interpersonal  needs 
(affective  requirements)  and  job  demands  (production  requirements)  is  usually  not  clear. 

Furthermore,  the  ability  to  apply  or  act  in  accordance  with  a  principle  often 
depends  upon  the  attitude  and  personality  of  the  individual  in  question.  For  Example,  an 
autocratic  person  may  find  it  impossible  to  act  in  a  desired  laissez-faire  or  democratic 
manner. 

Thus,  it  can  be  seen  that,  in  general,  social  skill  requires  problem  solving  ability.  The 
nature  of  specific  personal  and  situational  characteristics  must  be  identified  and  related  to 
general  principles  before  action  can  be  taken.  Such  ability,  like  the  development  of 
cognitive  skill  in  general,  is  only  acquired  through  practice  in  specific  situations.  Acquisi¬ 
tion  of  interpersonal  skill  probably  occurs  in  very  small  increments,  because  problem 
situations  do  not  completely  repeat  themselves  and  feedback  on  the  efficacy  of  a  solution 
is,  at  best,  imperfect.  More  practice  is  probably  required  for  the  development  of 
interpersonal  skill  than  for  any  of  the  preceding  varieties  of  skill. 

The  acquisition  of  social  skills  encompasses  the  learning  of  many  different  things. 
People  learn  to  give  and  take  criticism  and  direction.  They  learn  to  role  play  or  see  things 
from  another  person’s  point  of  view  and  to  overcome  strong  personal  dispositions.  They 
learn  certain  general  requirements  in  dealing  with  people;  individuals  must  be  shown 
tolerance  and  respect,  and  one  must  be  sensitive  to  the  feelings  and  needs  of  coworkers, 
superiors,  and  subordinates. 

Some  people  have  strong  needs  to  achieve,  to  be  liked,  to  be  dependent  upon  or 
dominate  others;  some  are  aggressive,  while  others  are  shy  and  retiring.  Social  skill  is 
dependent  upon  one’s  ability  to  recognize  and  accommodate  to  such  needs  in  others. 
People  learn  to  exert  a  degree  of  balance  or  focus  in  adjusting  their  behavior  in  response 
to  these  needs  in  accordance  with  specific  requirements  of  the  work  situation.  In  a  job, 
decisions  must  be  made  regarding  the  extent  to  which  a  coworker’s  personal  needs  should 
influence  behavior  and  the  extent  to  which  the  need  to  accomplish  specific  tasks  or  ends 
should  influence  behavior. 

Actions  will  vary  according  to  the  structure  of  the  social  situation.  When  one  is 
interacting  with  a  single  individual,  greater  emphasis  may  be  given  to  accommodating  to 
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his  particular  style  of  behavior.  When  a  group  of  people  is  involved,  other  considerations 
may  assume  greater  importance.  One  must  develop  the  ability  to  see  the  informal 
structure  of  a  group  and  to  perceive  social  reality  in  an  undistorted  fashion.  A  new 
person  entering  a  group,  whether  he  is  a  supervisor  or  another  worker,  needs  to  recognize 
the  natural  leader  of  the  group,  existing  friendship  patterns,  and  group  norms.  Problems 
arise  when  such  features  are  not  recognized  and  one  attempts  to  respond  solely  to  the 
formal  characteristics  of  a  group. 

Most  important,  a  person  must  develop  an  awareness  of  how  his  own  behavior  and 
manner  of  responding  affects  others,  whether  or  not  he  and  his  feelings  are  understood  as 
he  intends  them.  Thus,  a  supervisor  may  intend  to  create  an  atmosphere  of  understanding 
and  permissiveness  when,  in  fact,  he  has  accomplished  precisely  the  opposite.  In  some 
cases  the  absence  of  specific  guidance  may  prove  far  more  threatening  than  a  clearly 
structured  and  admittedly  autocratic  work  situation.  With  some  workers,  attempted 
permissiveness  may  be  interpreted  as  a  sign  of  weakness  and  a  lack  of  ability  on  the  part 
of  the  supervisor.  Such  misunderstanding  is  more  likely  to  arise  if  the  supervisor  does 
not,  in  reality,  want  to  play  such  a  role  and  does  not  do  it  well. 

A  person  learns  what  to  observe  or  look  for  in  interpersonal  behavior  and  how  to 
interpret  it.  Because  the  spoken  word  often  cannot  be  taken  at  face  value,  he  learns  to 
interpret  more  subtle  actions,  intonations,  and  modes  of  expression.  He  also  learns  to 
sense  or  see  through  biased  or  distorted  information  and  to  anticipate  signs  of  inter¬ 
personal  conflict  revealed  in  such  symptoms  as  tardiness,  avoidance  behavior,  and  critical 
comments. 

As  indicated,  practice  as  opposed  to  the  acquisition  of  information,  is  crucial  for  the 
development  of  social  skill.  Learning  how  others  respond  and  how  one  should  act  with 
others  in  order  to  produce  certain  desired  behavior  in  them  can  be  accomplished  only 
through  rather  extended  exposure  to  interpersonal  situations  and  problems.  Like  other 
skills,  social  skill  can  be  demonstrated  and  assessed  only  through  performance  in  inter¬ 
personal  situations. 


MOTIVATION  AND  PERSONALITY  CHARACTERISTICS 

The  terms  motivation  and  personality  characteristics  are  used  here  to  refer  to  a  class 
of  more  general  attitudes,  traits,  and  styles  of  behavior  that  condition  the  manner  in 
which  a  job  will  be  performed.  Included  in  this  class  are  such  characteristics  as  punc¬ 
tuality,  dependability,  ability  to  tolerate  frustration,  industriousness,  and  need  to  achieve. 

Successful  job  performance  requires  not  only  that  an  incumbent  possess  certain 
knowledge  and  skill,  but  that  he  possess  the  motivation,  the  desire,  and  the  willingness  to 
perform.  Different  levels  of  job  motivation  will  result  in  different  levels  of  job  perform¬ 
ance.  In  some  jobs,  heightened  motivation  may  compensate  in  part  for  deficits  in 
knowledge  and  skill.  Also,  certain  habitual  or  characteristic  ways  in  which  a  person 
generally  responds  to  persons  and  things  may  affect  his  job  performance  (personality). 
For  example,  a  person  who  typically  responds  to  other  persons  in  an  aggressive  manner  is 
not  likely  to  perform  well  in  a  supervisory  position. 

Although  job  motivation  and  personality  contribute  to  job  performance,  generally  it 
is  not  practicable  to  attempt  to  assess  such  factors  as  part  of  the  measurement  of  job 
knowledge  or  job  skill.  While  job  motivation  and  certain  personality  traits  may  affect 
performance  during  knowledge  and  skill  testing,  such  testing  provides  neither  an 
independent^  nor  a  reliable  estimate  of  these  factors.  A  man  may  lack  motivation  to 

^  Where  testing  is  undertaken  for  diagnostic  purposes,  independent  assessment  of  each  component 
of  performance  is  desirable. 
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work  at  his  job  and  may  perform  poorly  in  it,  but  he  may  be  highly  motivated  to 
perform  well  in  a  test  situation,  and,  consequently,  do  well  on  the  test.  Adequate 
assessment  of  motivation  and  personality  traits  is  more  effectively  accomplished  through 
peer  or  supervisor  ratings  based  on  continuous  observation  over  an  extended  period  of 
time.’ 


TYPES  OF  PROFICIENCY  MEASURES  AND  PURPOSES  OF  MEASUREMENT 
MEASURES  TO  ASSESS  JOB  PROFICIENCY 

The  criterion  instruments  used  in  the  UTILITY  research,  job  sample  tests,  job 
knowledge  tests,  and  supervisor  ratings,  may  be  identified  among  a  large  variety  of 
measures  that  can  be  used  to  assess  job  proficiency.  Thorndike  (12)  has  provided  an 
extensive  analysis  and  discussion  of  the  varieties  of  measures  that  can  be  used  for  this 
purpose.  An  outline  of  his  classification  of  measures  is  given: 

Evaluation  of  specific  defined  tasks 

Evaluation  of  knowledge  and  understanding 
Appraisals  of  actual  job  performance 
Objective  performance  score 
Observer  scored  job  samples 
Rated  job  samples 
Summary  evaluations 

Summary  performance  records 
Summary  academic  grades 
Summary  ratings 

Evaluation  of  defined  tasks  refers  to  the  assessment  of  a  limited  segment  of  job 
behavior.  It  includes  evaluation  of  knowledge  and  understanding  as  accomplished  in 
paper-and-pencil  job  knowledge  tests  (the  job  knowledge  tests  used  in  Work  Unit 
UTILITY  fall  into  this  class);  objective  performance  scores  based  upon  a  permanent  job 
record  or  product,  such  as  a  sample  of  typing  or  a  shot  pattern  on  a  rifle  target; 
observer-scored  job  samples,  such  as  a  performance  check  list  in  which  steps  in  a  job  task 
or  sequence  are  scored  as  having  been  performed  or  not  (the  job  sample  tests  used  in 
Work  Unit  UTILITY  fall  into  this  class);  and  rated  job  samples  in  which  the  adequacy  of 
the  performance  of  an  entire  task  is  rated  by  an  observer.  Summary  evaluations  include 
summary  performance  records  such  as  the  weekly  output  for  piece  workers  or  the  annual 
sales  record  of  a  salesman;  summary  academic  grades,  and  summary  ratings  such  as  a 
supervisor  or  peer  rating  of  an  employee  (the  supervisor  ratings  used  in  Work  Unit 
UTILITY  fall  in  this  class). 

While  a  discussion  of  the  characteristics,  advantages,  and  disadvantages  of  the 
different  types  of  measures  is  beyond  the  scope  of  the  present  report  (see  Thorndike,  1^, 
for  an  excellent  treatment),  it  is  readily  seen  that  the  suitability  of  a  particular  type  of 
measure  depends  partly  on  the  purpose  for  which  information  about  performance  is  being 
gathered.  Summary  evaluations  are  inadequate  for  providing  diagnostic  information  about 
a  student’s  strengths  and  weaknesses  during  training;  job  knowledge  tests  cannot  provide 
information  about  a  job  incumbent’s  skill,  motivation,  or  productivity,  and  so  on. 


’  Published  personality  tests  typically  show  a  low  relationship  between  measures  of  personality 
constructs  and  job  performance.  This  is  due,  in  part,  to  the  general  nature  of  such  traits,  as  measured. 
Ratings  of  the  specific  behavioral  characteristics  required  for  performance  in  particular  jobs  would 
appear  to  provide  a  more  promising  approach  to  the  measurement  of  personality  traits. 
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PURPOSES  OF  MEASUREMENT 


The  measurement  of  job  proficiency  can  serve  a  variety  of  purposes.  First,  pro¬ 
ficiency  testing  may  be  undertaken  to  assess  an  individual’s  performance  of  a  particular 
task  or  job  (i.e.,  to  provide  an  estimate  of  the  effectiveness  of  job  performance  or  likely 
performance).  Such  information  can  be  used  for  a  variety  of  purposes:  to  determine 
whether  a  trainee  is  eligible  for  graduation  at  the  end  of  training,  for  assigning  him  to  a 
particular  job  or  job  level,  and  for  establishing  his  suitability  for  advancement  and 
promotion  within  a  job. 

Measurement  of  performance  effectiveness  requires  assessment  of  the  components  of 
job  performance— knowledge,  skill,  motivation,  and  personality  characteristics.  To  evaluate 
knowledge,  a  job  knowledge  test  is  most  appropriate.  While  knowledge  can  be  evaluated 
by  means  of  job  sample  tests,  it  is  generally  extremely  inefficient  to  do  so.  However, 
where  the  knowledge  that  supports  performance  has  not  been  adequately  identified, 
appraisal  of  actual  job  performance,  such  as  job  sample  tests,  must  be  used.  To  evaluate 
skill,  one  of  the  varieties  of  job  sample  measures  should  be  used.  Objective  performance 
scores  or  job  sample  scores  are  to  be  preferred  to  the  more  subjective,  rated  job  samples. 
To  evaluate  motivation  and  personality  characteristics,  a  summary  rating  is  necessary.  As 
indicated,  published  personality  tests  have  generally  proven  unsatisfactory  for  this  pur¬ 
pose.  Rating  scales  for  assessing  motivation  should  be  tied  to  specific  observable  job 
behaviors  rather  than  being  based  upon  poorly  defined  general  impressions. 

Second,  proficiency  testing  may  be  undertaken  to  diagnose  performance  deficiencies 
either  during  training  or  on  the  job.  Such  testing  has  a  pedagogical  purpose  only— to 
determine  whether  the  trainee  has  or  has  not  acquired  specific  knowledge  and  skill 
necessary  to  support  performance  and  to  permit  the  adjustment  of  training  to  rectify 
deficiencies. 

To  diagnose  knowledge  deficiencies,  job  knowledge  tests  are  most  appropriate.  For 
this  purpose,  it  is  usually  more  efficient  and  informative  to  reduce  all  knowledge 
requirements  to  their  simplest  components.  A  series  of  multiple-choice  items  measuring 
the  separate  knowledges  required  for  the  accomplishment  of  a  complex  task  is  to  be 
preferred  to  a  single  item  that  attempts  to  present  simultaneously  the  full  variety  of 
stimulus  possibilities  the  trainee  would  face  in  the  performance  of  the  actual  job.  Such  an 
item  provides  only  a  global  measure  of  the  availability  of  many  knowledges. 

To  diagnose  skill  deficiencies,  observer-scored  job  samples,  such  as  job  sample  tests, 
are  most  appropriate.  Objective  performance  scores  based  upon  a  job  record  or  product 
will  usually  not  provide  information  in  sufficient  detail  for  diagnostic  purposes.  While 
rated  job  samples  can  be  used,  they  are  less  preferred  because  of  their  subjective 
component. 

Third,  proficiency  testing  may  be  used  to  supplement  the  teaching  process.  The 
particular  virtue  of  requiring  a  trainee  to  demonstrate  information  and  skill  in  a  test  lies 
in  the  strong  motivating  properties  of  the  testing  situation.  Individuals  who  know  that 
they  will  be  tested  may  be  expected  to  concentrate  on  learning  materials  and  skills  more 
thoroughly.  Of  the  measures  outlined  by  Thorndike  (^),  evaluation  of  specific  defined 
tasks  that  can  be  attained  at  any  stage  of  training  are  probably  most  appropriate  for 
motivational  purposes. 

Fourth,  proficiency  testing  may  be  undertaken  to  evaluate  the  effectiveness  of  a 
training  program.  The  development  of  a  training  program  logically  begins  with  the 
specification  of  certain  performance  capabilities  that  trainees  are  to  acquire.  Training 
content  (skills  and  knowledge  to  be  acquired),  method,  and  organization  are  then 
developed  to  operate  collectively  in  providing  trainees  going  through  the  program  with 
the  desired  capabilities.  When  training  development  has  been  completed,  graduates  of  the 
program  are  tested  to  determine  the  extent  to  which  these  goals  have  been  met. 
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Once  the  adequacy  of  a  training  program  has  been  established  through  such  an 
evaluation  process,  individuals  who  successfully  complete  the  various  subparts  of  the 
training  program  can  be  presumed  to  possess  the  desired  capabilities.  While  end-of-training 
testing  may  still  be  conducted,  such  testing  is  no  longer  essential  to  ensure  adequacy  of 
the  training  product  except  as  it  may  be  used  to  determine  the  continuing  quality  of  the 
program.  To  establish  the  adequacy  of  a  training  program  (i.e,,  program  evaluation),  one 
of  the  various  methods  for  appraising  job  performance  is  essential.  Again,  objective 
performance  scores  or  job  sample  tests  are  preferred  over  rated  job  samples. 

Fifth,  proficiency  testing  may  be  undertaken  to  provide  a  criterion  measure  in 
research.  Thus,  the  testing  in  Work  Unit  UTILITY  was  undertaken  primarily  to  determine 
the  job  proficiency  of  men  at  different  ability  (AFQT)  levels.  Another  frequent  research 
purpose  for  proficiency  tests  is  to  validate  other  measures  that  are  more  remote  from  the 
ultimate  criterion  (actual  job  performance),  but  may  be  more  convenient,  economical, 
and  suitable  for  operational  use  (13). 

In  addition  to  estimating  the  job  proficiency  of  men  in  different  ability  groupings, 
the  data  obtained  in  UTILITY  may  be  examined  to  provide  information  about  the 
empirical  relationships  among  the  three  criterion  measures. 

DESCRIPTION  OF  THE  UTILITY  TESTS 

Job  sample  tests  were  used  as  the  closest  approximation  possible  to  the  ultimate 
criterion  of  actual  job  performance.  Each  job  sample  test  comprised  a  number  of 
subtests,  was  individually  administered,  and  took  from  three  and  one-half  to  five  hours  to 
complete.  To  make  testing  realistic  and  to  simulate  job  conditions  with  as  much  fidelity 
as  possible,  each  subtest  consisted  of  the  performance  of  a  single  entire  task  with  a 
natural  beginning  and  ending.  Tasks  were  composed  of  a  series  of  actions  or  steps  that 
would  ordinarily  be  performed  as  part  of  a  single  operation  in  the  performance  of  a  job. 

Subtest  tasks  or  problems  were  introduced  to  subjects  just  as  they  might  ordinarily 
encounter  them.  For  example,  one  of  the  Repairman  tasks  involved  the  diagnosis  and 
correction  of  a  leaking  oil  seal  in  a  fan  shaft  of  a  tank.  The  tank  was  prepared  so  that 
the  oil  seals  leaked  and  the  subject  was  told  that  oil  was  being  thrown  out  of  the  tank’s 
grill  doors.  The  Repairman  was  to  locate  the  source  of  the  oil  leak  and  to  repair  the 
malfunction.  In  this  problem,  there  were  19  necessary  steps  (Figure  1).  Subjects  earned  a 
point  for  the  correct  performance  of  each  step.  This  scoring  procedure  was  used  in  all  job 
sample  tests. 

The  subtests  contained  in  the  four  job  sample  tests  are  identified  in  Table  1. 

In  UTILITY,  job  knowledge  (paper-and-pencil)  tests  were  used  to  supplement  job 
sample  tests  and  provide  an  efficient  means  of  sampling  from  the  entire  range  of 
knowledge  required  in  job  performance.  These  tests,  using  a  multiple-choice  format,  were 
similar  to  those  traditionally  used  to  assess  learning  in  classroom  situations  and  in  Army 
tredning.  However,  in  the  development  of  these  tests,  particular  care  was  taken  to  ensure 
that  only  the  knowledge  needed  to  support  job  performance  was  covered.  Job-related 
information  that  is  not  required  to  support  job  performance  has  often  been  used  to 
provide  better  discrimination  among  job  incumbents,  but  it  is  invalid  if  the  purpose  of 
the  test  is  to  estimate  job  proficiency.  A  few  illustrative  items>  taken  from  the  Supply 
Specialist  job  knowledge  test,  are  given  in  Figure  2. 

Supervisor  ratings  were  obtained  for  assessing  motivation  and  other  personal  attri¬ 
butes,  which  the  job  incumbent  had  characteristically  demonstrated  in  the  performance 
of  his  job.  This  instrument  consisted  of  the  14  rating  scales  that  make  up  the  Enlisted 
Efficiency  Report  used  operationally  throughout  the  Army.  In  Figure  3,  two  examples 
from  the  Supervisor  Rating  Scales  are  presented. 
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Steps  in  Fan  Vertical  Shaft  Oil  Seal  Replacement  Subtest— 

Repairman  Job  Sample  Test 

1 .  Testee  removes  a  fan  vane  by  removing  four  screws  securing  vane  to  housing. 

2.  Removes  cotter  pin,  slotted  hex  nut,  and  flat  washer,  securing  fan  assembly  to  fan  drive  shaft. 

3.  Removes  fan  assembly  from  shaft. 

4.  Reports  fan  tower  seals  leaking. 

5.  Cuts  and  removes  locking  wire  from  bolts  in  oil  seal  housing. 

6.  Removes  six  capscrews  securing  housing. 

7.  Screws  two  5/16-18  X  4  bolts  in  threaded  holes  in  oil  seal  housing. 

8.  Turns  both  bolts  in  evenly  until  oil  seal  housing  separates  from  fan  drive  housing  cover. 

9.  Removes  bolts  from  oil  seal  housing. 

(Tester  says:  ^TOR  THE  PURPOSE  OF  THIS  TEST;  WE  WILL  NOW  CONSIDER  THAT  YOU  HAVE 
REPLACED  BOTH  DEFECTIVE  SEALS.  HOW  WOULD  YOU  REPLACE  THE  SEALS?" 

10.  Testee  answers  "Remove  old  seal,  clean  housing,  coat  housing  with  adhesive  cement,  and 
install  new  seal." 

(Tester  says:  "NOW,  YOU  MUST  PUT  THIS  VEHICLE  IN  SERVICEABLE  CONDITION.") 

1 1.  Secures  oil  seal  housing  with  six  bolts  and  flat  washers. 

12.  Installs  locking  wire  through  bolt  heads. 

13.  Replaces  fan  assembly  on  splined  drive  shaft. 

14.  Replaces  flat  washer  on  shaft. 

15.  Replaces  slotted  hex  nut  on  shaft. 

16.  Torques  nut  to  50  Ib-ft  torque  and  aligns  slot  with  hole  in  shaft. 

17.  Inserts  new  cotter  pin  through  nut  and  shaft. 

18.  Checks  clearance  between  each  fan  blade  and  fan  housing  with  0.070-inch  feeler  gauge.  Adjusts 
fan  housing  as  necessary. 

19.  Installs  unmarked  fan  vane  housing  over  rear  fan  or  installs  fan  vane  housing  marked  "Damper 
End"  over  front  fan. 

Figure  1 
Table  1 


Job  Sample  Test 


Title  and 
Subtest  Number 

Description 

Number  of 
Steps 

Armor  Crewman 

1 

Disassembly  and  Assembly  of  the  Cal.  45  Pistol, 

2 

M1811A1 

Disassembly  and  Assembly  of  the  Cal.  45  Submachine 

28 

3 

Gun,  M3A1 

Disassembly  and  Assembly  of  the  Coaxial  Machine  Gun, 

30 

4 

M73 

Disassembly  and  Assembly  of  the  Cal.  50  Machine  Gun, 

23 

5 

M85 

Identification  of  Arm  and  Hand  Flag  and  Light 

45 

6 

Signals 

Identification  of  Combat  Formations  for  Tanks 

24 

10 

7 

Mounting  and  Operating  the  AN/VRC  53  Tank  Radio 

10 

(Continued) 
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Table  1  (Continued) 

Job  Sample  Test 


Title  and 
Subtest  Number 


Description 


Number  of 
Steps 


Armor  Crewman 
(Cont.) 

8  Identifying  and  Stating  the  Functions  of  the  M60A1 

Tank  Controls  16 

9  Starting  and  Stopping  the  Main  Engine  in  Moderate, 

Hot,  and  Cold  Weather  19 

10  Driving  the  M60A1  Tank  in  Response  to  Arm  and 

Hand  Signals  9 

11  Before  Operations  Maintenance  of  the  M60A1  Tank  15 

12  Mounting  and  Operating  the  AN/VRC  12  Tank  Radio  14 

13  Using  the  MlAl  Gunner's  Quadrant  and  Adjusting  the 

Elevation  Quadrant,  M13A3  15 

14  Preparation  of  Range  Card  15 

15  Loading  the  105-mm  Gun,  M68  7 

16  Reading  and  Setting  the  Ballistic  Computer  22 

17  Boresighting  and  Zeroing  the  105*mm  Gun  17 

18  Checking  the  Azimuth  Indicator  for  Accuracy  and 

Slippage,  and  Putting  the  Turret  Into  Power  20 

19  Adjustment  of  Fire  in  Response  to  Fire  Commands  9 

20  Reading  and  Setting  the  Azimuth  Indicator  6 

Repairman 

1  Transmission  Shift  Linkage  Adjustment  (M60A1  Tank)  11 

2  Transmission  Servo  Band  Adjustment  (M60A1  Tank)  13 

3  Voltage  Regulator  Adjusting  Rheostat  Adjustment 

(M60A1  Tank)  16 

4  Hydraulic  Brake  Pedal  Adjustment  (M60A1  Tank)  11 

5  Fan  Vertical  Drive  Shaft  Oil  Seal  Replacement 

(M60A1  Tank)  19 

6  Faulty  Spark  Plug  Check  (M151  Truck)  9 

7  Ignition  Timing  Adjustment  (M151  Truck)  19 

8  Cylinder  Compression  Check  (Ml 51  Truck)  13 

9  Battery  Hydrometer  Check  (M151  Truck)  9 

10  Battery  Condition  Test  (M151  Truck)  15 

11  Front  Outer  Wheelbearing  Adjustment  13 

12  Fuel  Pump  Pressure  Check  (M35A1/A2  or  M49C  Truck)  10 

13  Brake  Bleeding  (M35A1/A2  or  M49C  Truck)  17 

Supply  Specialist 

1  Preparation  of  Laundry  Roster  25 

2  Preparation  of  Personal  Clothing  Request  15 

3  Preparation  of  Hand  Receipts  12 

4  Sorting  Equipment  for  Serviceability  10 

5  Preparation  of  Maintenance  Request  27 

6  Preparation  of  Statement  of  Charges  33 

7  Preparation  of  Request  for  Issue  19 

8  Maintenance  of  Document  Register  15 

- (Continued) - 


Table  1  (Continued) 

Job  Sample  Test 


Title  and 
Subtest  Number 

Description 

Number  of 
Steps 

Cook 

1 

Lighting  the  Fire  Unit  of  the  M1937  Field  Range 

29 

2 

Preparation  of  Cocoa  with  Reconstituted  Milk 

38 

3 

Preparation  of  Scrambled  Eggs 

10 

4 

Preparation  of  Jelly  Roll 

45 

5 

Preparation  of  Cook's  Worksheet 

36 

Illustrative  Items  From  the  Supply  Specialist  Job  Knowledge  Test 

6.  What  is  a  Document  Register  used  for? 

A.  Filing  hand  receipts 

B.  Issuing  documents 

C.  Recording  document  numbers 

D.  Recording  unserviceable  equipment 

12.  What  form  is  used  to  request  line  item  expendable  supplies? 

A.  DA  Form  2765-1 

B.  DA  Form  2765 

C.  DA  Form  3161 

D.  DA  Form  2064 

20.  What  regulation  governs  the  issue  and  sale  of  personal  clothing? 

A.  AR  735-502-3 

B.  AR  700-8400-1 

C.  AR  712-320-5 

D.  AR  725-220-3 

38.  What  step  is  taken  first  when  you  receive  the  Equipment  Status  Report? 

A.  The  Company  Commander  signs  the  report 

B.  You  edit 

C.  Increase  the  on-hand  quantity 

D.  You  make  an  issue 

66.  You  want  to  check  to  see  if  the  forms  you  are  using  are  current.  What  DA  pamphlet  index 
would  you  refer  to? 

A.  310-1 

B.  310-2 

C.  310-3 

D.  310-4 

88.  When  a  recorded  serial  numbered  item  has  been  turned  in  how  is  it  posted  on  the  back  of 
the  Property  Book  page? 

A.  Entered  as  a  turn-in 

B.  Lined  out  and  initialed 

C.  Debit  posted  and  initialed 

D.  Posted  as  a  gain 

Figure  2 


Illustrative  Items  From  the  Supervisor  Rating  Scales 
(Enlisted  Efficiency  Report) 

ACCEPTABILITY  (Ability  to  get  along  well  with  others) 

a.  _ Never  gets  along  well  with  other  people 

b.  _ Sometimes  does  not  fit  into  group 

c.  _ Gets  along  fairly  well  with  others 

d.  _ Gets  along  well  with  most  people 

e.  _ Gets  along  very  well  with  others 

f.  _ Gets  along  exceptionally  well  with  others 

RELIABILITY  (Dependability  and  consistency  of  performance) 

a.  _ Is  not  dependable 

b.  _ Is  sometimes  late  in  finishing  jobs 

c.  _  Is  average  in  reliability 

d.  _ Works  steadily 

e.  _ Can  always  be  relied  upon  to  sticK  to  the  job 

f.  _ Shows  exceptionally  dependable  performance 

Figure  3 


In  addition  to  the  two  examples  given  in  Figure  3,  the  following  characteristics  were 
rated: 

(1)  Conduct  (ability  to  control  and  manage  personal  behavior). 

(2)  Job  performance  (achievement  of  job  tasks). 

(3)  Cooperativeness  (ability  to  work  effectively  with  others). 

(4)  Drive  (energy  devoted  to  job  mastery  and  self-improvement). 

(5)  Development  (effort  directed  toward  realization  of  potential). 

(6)  Initiative  (aggressive  pursuit  of  methods  to  improve  performance  or 
productivity). 

(7)  Job  knowledge  (effort  directed  toward  improvement  of  job  knowledge). 

(8)  Application  (willingness  to  work). 

(9)  Participation  (contribution  to  group  productive  effort). 

(10)  Adaptability  (ability  to  perform  effectively  in  the  face  of  changing  job 
demands). 

(11)  Responsibility  (willingness  to  accept  responsibility). 

(12)  Leadership  (ability  to  direct  the  work  of  others).’ 


AN  ANALYSIS  OF  THE  INTERCHANGEABILITY  OF 
PROFICIENCY  MEASURES  AND  EVIDENCE  FROM  THE  UTILITY  DATA 

When  proficiency  testing  is  undertaken  to  estimate  the  effectiveness  of  job  per¬ 
formance,  a  combination  of  knowledge,  skill,  and  motivational  testing  is  often  required. 
Nevertheless,  practical  constraints  can  be  expected  to  promote  the  use  of  only  those 
criterion  measures  that  are  simplest  and  most  economical  to  administer.  It  is  essential, 
therefore,  to  determine  where  one  criterion  instrument  can  reasonably  be  substituted  for 
another. 

'  Detailed  descriptions  of  each  instrument  and  its  development  and  administration  are  provided  in 
the  second  report  of  this  series  (2). 
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An  instrument  that  has  been  designed  to  assess  job  motivation  and  other  personal 
qualities,  such  as  rating  scales,  is  always  essential  if  a  complete  estimate  of  job  perform¬ 
ance  is  desired.  These  results  would  be  examined  in  conjunction  with  more  direct 
measures  of  job  skill  and  job  knowledge.  The  decision  as  to  the  appropriateness  of  using 
a  job  sample  test  alone,  a  job  knowledge  test  alone,  or  both,  depends  upon  the  weight  of 
the  skill  and  knowledge  components  in  a  job;  this  can  be  determined  only  after  a  careful 
analysis  and  specification  of  the  information  and  skill  needed  to  support  performance. 

Where  job  performance  is  mediated  almost  solely  by  knowledge  per  se,  job 
knowledge  tests  should  be  used  to  assess  the  information  a  man  has  about  his  job.  Job 
sample  tests  should  not  be  used  in  such  a  situation  since  they  are  far  more  expensive  to 
administer. 

Where  job  performance  relies  almost  solely  on  skill,  job  sample  tests,  or  some  other 
variety  of  performance  measure,  are  essential.  A  convenient  means  for  deciding  on  the 
skill  requirements  of  a  job,  and  hence  the  extent  to  which  performance  testing  is 
necessary,  is  to  determine  how  much  practice  or  rehearsal  is  typically  needed  during 
training.* 

Where  job  performance  depends  upon  a  mix  of  knowledge  and  skill,  both  knowledge 
tests  and  performance  tests  are  indicated.  Thus,  to  determine  whether  a  person  should  be 
given  an  automobile  driver’s  license,  most  states  administer  both  a  paper-and-pencil  (job 
knowledge)  test  to  assess  knowledge  about  driving  and  a  performance  (job  sample)  test  to 
assess  driving  skill  or  ability.  Both  types  of  test  are  necessary  because  each  alone  will  not 
provide  sufficient  information  about  a  license  applicant’s  capability  to  warrant  issuing 
him  a  license.  The  knowledge  test  cannot  reveal  whether  he  has  skill  enough  to  drive  an 
automobile,  and  the  driving  test  cannot,  v^thout  great  expense,  be  made  extensive 
enough  to  reveal  his  knowledge  of  driving  rules,  road  signs,  laws  regulating  vehicles,  and 
so  forth. 

On  the  other  hand,  the  ability  of  a  clerk  to  perform  his  job  is  almost  entirely 
dependent  upon  his  possession  of  certain  knowledge  and  information.  The  adequacy  with 
which  he  makes  entries  in  documents,  and  files  material,  can  be  established  readily 
through  knowledge  testing  (14).  A  job  sample  test  is  not  necessary  unless  speed  and 
neatness  are  critical;  then  a  job  sample  test  would  also  be  necessary. 

Job  sample  testing  is  expensive  in  time,  money,  development,  and  test  adminis¬ 
trators;  therefore,  reliance  on  job  knowledge  tests  (and  ratings  of  motivation  and  other 
personal  characteristics)  to  assess  job  proficiency  is  desirable,  if  valid  measurement  can  be 
accomplished  in  this  manner.  Yet,  the  correlations  between  job  knowledge  test  scores  and 
job  sample  criterion  have  often  been  found  to  be  too  low  to  support  the  use  of  job 
knowledge  tests  alone  for  assessing  proficiency  (13,  1^,  ^).  Nevertheless,  job 

knowledge  tests  should  be  valid  for  measuring  proficiency  in  all  jobs  where  skill  com¬ 
ponents  are  minimal,  if  job  knowledge  tests  are  carefully  constructed  to  measure  only 
information  that  is  directly  relevant  for  performing  the  job.  Job  knowledge  tests  fre¬ 
quently  correlate  poorly  with  performance  criteria  because  they  include  information  that 
is  irrelevant  to  job  performance.  For  example,  some  varieties  of  theoretical  information, 
terminology,  and  historical  information,  commonly  found  in  training  or  course  content, 
are  readily  available  for  the  construction  of  test  items.  Because  such  items  are  often  easy 


*  It  should  be  noted  that  a  distinction  has  been  made  between  skill  that  requires  practice  or 
rehearsal  during  training  and  skill  that  the  trainee  possesses  at  the  outset  of  training  and  for  which 
practice  and  rehearsal  are  not  necessary.  For  example,  there  are  skills  that  males  usually  acquire  while 
growing  up  in  our  culture,  such  as  the  use  of  simple  tools  (screw  driver,  wrench,  hammer,  etc.).  In 
general,  such  skill  requirement  can  be  overlooked  as  far  as  testing  is  concerned,  if  the  trainee  or  job 
incumbent  population  can  be  assumed  to  possess  the  skill.  However,  if  the  population  were  to  change, 
such  an  assumption  would  have  to  be  reconsidered. 
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to  prepare,  they  are  frequently  overrepresented  in  tests  even  though  they  do  not  bear 
directly  upon  performance  of  the  job. 

An  analysis  of  the  job  sample  tests  used  to  measure  performance  in  the  four  jobs 
studied  in  UTILITY  showed  that  skill  requirements  in  these  jobs  appear  to  be  minimal. 
Table  2  shows  the  number  of  steps  in  each  job  sample  test  that  were  judged  to  be 
mediated  by  knowledge  alone,  and  those  judged  to  require  cognitive  skill  or  some 
combination  of  perceptual  and  motor  skill.  None  of  the  steps  in  any  of  the  performance 
tests  required  social  skill.*  Practically  all  of  the  performance  steps  in  each  of  the  jobs  can 
be  supported  by  knowledge  alone,  and  as  was  anticipated,  skill  requirements  in  the 
Supply  Specialist  job  are  virtually  nonexistent. 

This  analysis  suggests  that  the  correlation  between  job  sample  test  scores  and  job 
knowledge  test  scores  in  each  of  these  jobs  would  be  high,  and,  in  all  likelihood,  the 
correlation  would  be  highest  in  the  Supply  Specialist  job. 

Table  2 

Frequency  of  Steps  in  Job  Sample  Tests  Categorized 
According  to  Knowledge  and  Skill  Requirements 


Requirements 

Armor 

Crewman 

Repairman 

Supply 

Specialist 

Cook 

Knowledge  Alone 

338 

165 

153 

145 

Cognitive  Skill  and  Knowledge 

0 

4 

3 

11 

Perceptual -Motor  Skill  and 
Knowledge 

16 

6 

0 

2 

Total  Number  of  Steps 

354 

175 

156 

158 

INTERRELATIONSHIPS  OF  JOB  SAMPLE  TESTS, 

JOB  KNOWLEDGE  TESTS,  AND  SUPERVISOR  RATINGS 

Correlations  among  job  sample  scores,  job  knowledge  scores,  and  supervisor  ratings 
are  given  in  Table  3.  Correlations  between  these  scores  are  presented  for  the  combined 
Category  IV  and  Non-IV  groups.  While  the  combined  sample  is  artificial  (i.e.,  not 
representative  of  the  proportion  of  Category  IVs  to  Non-IVs  in  the  Army),  the  correla¬ 
tions  for  each  group  separately  are  so  similar  (Table  4)  that  the  discussion  of  the 
implications  of  these  correlations  has  been  simplified  by  using  the  combined  group 
correlation. 

Here,  as  in  an  earlier  report  in  this  series  (3),  zero-order  and  partial  correlations  are 
both  reported.  The  partial  correlations  provide  indices  of  the  strength  of  relationship 
between  the  criterion  measures  when  the  variance  that  each  holds  in  common  with  job 
experience  (months  on  the  job)  has  been  removed.  Both  the  zero-order  and  partial 


>\ 

'  A  few  of  the  men  tested  in  all  the  jobs  held  supervisory  positions,  undoubtedly  requiring;  .some 
degree  of  social  skill.  It  has  been  anticipated  however,  that  a  test  of  supervisor  skill  would  be 
inappropriate  for  the  majority  of  the  sample  and  a  decision  was  made  early  in  the  research  not  to 
attempt  to  measure  this  aspect  of  performance. 
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Table  3 


Correlations  of  Job  Sample,  Job  Knowledge,  and 
Supervisor  Ratings  for  the  Entire  Sample® 


Item 

Armor 

Crewman 

(N=368) 

Repairman 

(N*360) 

Supply 

Specialist 

(N=380) 

Cook 

(N-366) 

Zero-Order  Correlations 

Job  Sample— Job  Knowledge 

.68 

.59 

.72 

.58 

Job  Sample— Supervisor  Ratings 
Job  Knowledge— Supervisor 

.27 

.20 

.28 

.28 

Ratings 

.31 

.23 

.32 

.35 

Partial  Correlations^ 

Job  Sample— Job  Knowledge 

.49 

.49 

.65 

.50 

Job  Sample— Supervisor  Ratings 
Job  Knowledge— Supervisor 

.13 

.16 

.24 

.20 

Ratings 

.21 

.18 

.29 

.27 

®AII  correlations  are  significantly  different  from  zero  (p'^.OS). 
^Correlations  with  the  effects  of  time  on  the  job  (MOJ)  partlalled  out. 


Table  4 

Job  Sample  Test  Reliabilities  (Lower  Limit) 


Job  Title 

r 

Cat-IV 

r 

Non-IV 

r 

Combined 

Armor  Crewman 

.52 

.55 

.55 

Repairman 

.46 

.38 

.42 

Supply  Specialist 

.51 

.58 

.54 

Cook 

.36 

.34 

.35 

corrolations  show  a  high  degree  of  relationship  between  job  sample  and  job  knowledge 
test  scores,  but  a  considerably  lower  degree  of  relationship  between  either  of  these 
criteria  and  supervisor  ratings. 

The  correlations  obtained  here  between  job  sample  and  job  knowledge  test  scores 
are  higher  than  those  reported  elsewhere  (1^)  and  tend  to  support  the  authors’ 
hypotheses  that  job  knowledge  tests  can  be  appropriately  substituted  for  job  sample  tests, 
when  a  job  contains  little  or  no  skill  components  and  when  only  knowledge  required  on 
the  job  is  used  in  the  test. 

During  the  preceding  analysis  of  the  job  sample  test  step  requirements  (Table  2),  the 
skill  components  in  these  four  jobs  were  judged  to  be  minimal.  Even  in  the  Repairman 
and  Cook  jobs,  where  manual  operations  and  the  use  of  tools  are  common,  skill 
requirements  appeared  to  be  small.  It  may  be  assumed  that  most  persons  are  able  to 
perform  in  either  of  these  jobs  almost  completely  on  the  basis  of  knowledge  of  the 
required  tasks  and  of  the  specific  operations  entailed.  The  magnitude  of  the  correlations 
betwiHMi  job  sample  and  job  knowledge  scores  in  all  the  jobs  support  this  analysis.  As 
anticipated,  the  highest  correlation  occurred  between  job  sample  and  job  knowledge 
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criteria  in  the  Supply  Specialist  job.  This  relationship  is  of  particular  significance  for  this 
analysis.  A  clerk’s  job,  typified  by  the  Supply  Specialist,  represents  one  of  the  purest 
examples  of  a  job  where  knowledge  rather  than  skill  is  sufficient  to  support  performance. 

If  job  knowledge  tests  are  to  be  used  to  estimate  job  proficiency,  the  reliability  of 
these  measures  must  also  be  considered.  Reliability  of  job  knowledge  tests  was  obtained 
using  an  odd-even  split  of  test  items.  Reliability  coefficients,  for  both  Category  IVs  and 
Non-IVs  in  each  MOS  are  presented  in  Table  5.  Both  the  computed  reliability  based  upon 
the  odd-even  halves  and  the  reliabilities  corrected  for  test  length  (Spearman-Brown 
prophecy  formula)  are  given. 

Table  5 


Job  Knowledge  Test  Reliabilities 
(Correlations  Between  Odd  and  Even  Numbered  Items) 


Job  Title 

Category 

Computed  r 

r  Corrected 
for  Test  Length 

df 

Armor  Crewman 

Cat-IV 

.70 

.82 

189 

Non-IV 

.65 

.79 

189 

Combined 

.81 

Repairman 

CaMV 

.61 

.76 

193 

Non-IV 

.63 

.77 

193 

Combined 

.76 

Supply  Specialist 

Cat-IV 

.88 

.94 

198 

Non-IV 

.80 

.89 

198 

Combined 

.92 

Cook 

CaMV 

.69 

.82 

189 

Non-IV 

.74 

.85 

189 

Combined 

.84 

The  high  reliabilities  obtained  for  the  job  knowledge  tests  are  easily  sufficient  to 
permit  the  application  of  such  tests  in  situations  where  group  measurement  is  required, 
as,  for  example,  in  the  UTILITY  research.  Further,  these  reliabilities  appear  to  be  of 
sufficient  magnitude  to  allow  the  tests  to  be  used  in  the  measurement  of  individuals. 

The  lower  limits  of  the  reliabilities  of  the  job  sample  tests  were  estimated  using  the 
correlations  between  job  sample  and  job  knowledge  test  scores  and  the  reliabilities 
computed  for  the  job  knowledge  test  scores 

r^y  max  =  VT  ^xx)  (^yy) 

and  solving  the  equation  for  the  unknown  job  sample  reliability.*  These  estimates  of  the 
lower  limits  of  the  job  sample  test  reliabilities  are  presented  in  Table  4. 

*  It  was  not  appropriate  to  directly  compute  reliabilities  for  job  sample  tests  for  the  following 
reasons:  (a)  Job  sample  subtests  were  heterogeneous  (i.e.,  they  measured  different  skills  of  varying 
degrees  of  complexity  and  difficulty).  Under  these  conditions,  test-retest  estimate  of  reliability  is 
appropriate  rather  than  some  estimate  of  internal  consistency.  No  retest  data  were  possible  in  this  study, 
(b)  In  general,  within  any  subtest  steps  were  not  independent,  (c)  Not  all  subjects  performed  all  steps  (if 
a  subject  failed  to  complete  three  successive  steps,  the  subtests  were  discontinued). 
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An  examination  of  the  correlations  between  supervisor  ratings  and  the  two  test 
criteria  (Table  3)  shows  relatively  weak  relationships;  this  is  as  expected.  The  supervisor 
ratings  were  designed  to  measure  general  behavioral  characteristics  such  as  adaptability, 
reliability,  drive  and  conduct— a  composite  of  attributes  that  underlie  general  work  and 
interpersonal  effectiveness  (19).  This  instrument  was  included  in  UTILITY  in  order  to 
obtain  estimates  of  these  attributes  that  were  not  obtained  through  job  sample  or  job 
knowledge  testing. 

As  seen,  there  is  some  degree  of  relationsnip  between  supervisor  ratings  and  the 
other  two  criterion  measures.  Two  of  the  14  scales  in  the  supervisor  rating  provide 
specific  ratings  of  job  knowledge  and  job  performance,  thereby  making  a  direct  though 
small  contribution  to  the  relationships.  Also,  most  of  the  personal  attributes  would  be 
expected  to  contribute  to  the  acquisition  of  specific  knowledge  and  skill.  Thus,  there  is 
some  amount  of  correlation  between  supervisor  ratings  and  the  other  two  criteria. 
Nevertheless,  the  overall  relationship  between  supervisor  ratings  and  the  two  test  instru¬ 
ments  are  low,  since  the  former  is  also  measuring  attributes  other  than  job  knowledge 
and  job  performance  per  se.* 


RELATIONSHIPS  BETWEEN  JOB  SAMPLE  TESTS  AND 
JOB  KNOWLEDGE  TESTS  IN  SELECTED  SUBSAMPLES 

Having  observed  the  relatively  high  correlations  between  job  sample  and  job  knowl¬ 
edge  test  scores  for  the  entire  UTILITY  sample,  it  is  appropriate  to  determine  whether  or 
not  this  relationship  is  maintained  in  particular  subgroups.  It  is  possible  that  two  variables 
may  correlate  differently  in  a  particular  subgroup  than  in  the  overall  sample.  Were  there 
to  be  a  significantly  lower  correlation  between  job  sample  and  job  knowledge  scores  for  a 
subgroup,  it  would  be  inappropriate  to  substitute  job  knowledge  tests  for  job  sample  tests 
as  a  measure  of  job  proficiency  for  that  subgroup  (without  making  adjustment  to  take 
such  differences  into  account). 

Scores  on  job  knowledge  tests  tend  to  be  highly  related  to  reading  ability  (20). 
Therefore,  it  is  appropriate  to  examine  the  correlations  of  the  two  test  criteria  in 
subgroups  with  low  and  high  reading  ability,  and  in  subgroups  that  differ  in  charac¬ 
teristics  related  to  general  verbal  ability  to  determine  whether  the  relationships  between 
job  sample  and  job  know  edge  tests  are  maintained  in  these  groups.^  Category  I  Vs,  men 
with  low  verbal  aptitude  and  men  with  low  education,  were  all  found  to  have  signifi- 
czintly  lower  reading  ability  (^).  Blacks  in  the  sample  also  tended  to  have  lower  reading 
ability  (^).  Correlations  between  job  sample  and  job  knowledge  scores  for  the  entire 
sample  are  compared  with  ^hose  for  men  with  low  and  high  reading  ability  (Table  6), 
Category  IVs  and  Non-IVs  (Table  7),  men  with  low  and  high  verbal  aptitude  (Table  8), 
men  with  low  and  high  education  levels  (Table  9),  and  Blacks  and  Whites  (Table  10). 


*The  conditions  under  which  the  UTILITY  study  was  conducted  did  not  permit  inter-rater 
reliability  to  be  obtained  for  the  supervisor  ratings.  Reliabilities  based  upon  at  least  two  raters  would 
provide  the  most  meaningful  measures  of  stability  of  these  instruments.  Estimates  of  the  lower  limit  of 
supervisor  rating  reliabilities,  obtained  in  the  same  manner  as  those  computed  for  job  sample  tests, 
would  be  inappropriate  and  expected  to  provide  gross  underestimates  of  reliability. 

^Reading  ability  was  assessed  using  the  California  Test  Bureau  Survey  of  Reading  Achievement, 
Junior  High  Level  (grades  7-9);  verbal  ability  was  assessed  using  the  Verbal  Test  of  the  Army 
Classification  Battery;  educational  level  was  obtained  from  a  questionnaire  item:  Years  of  Education 
Completed.  Low  and  high  groups  were  obtained  for  each  of  these  variables  by  assigning  people  to  groups 
on  or  above  the  median  and  below  the  median. 
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Table  6 


Correlations  of  Job  Sample  and  Job  Knowledge 
for  Low  Reading  and  High  Reading  Ability  Subgroups^ 


Sample 

Armor 

Crewman 

Repairman 

Supply 

Specialist 

Cook 

n" 

r 

N 

r 

N 

r 

N 

r 

Zero-Order  Correlations 


Entire  Sample 

368 

.68 

360 

.59 

380 

.72 

366 

.58 

Low  Reading 

178 

.65 

174 

.45 

186 

.67 

176 

.42 

High  Reading 

190 

.57 

186 

.47 

194 

.64 

190 

.54 

Partial  Correlations^ 

Entire  Sample 

368 

.49 

360 

.49 

380 

.65 

366 

.50 

Low  Reading 

178 

.34 

174 

.44 

186 

.59 

176 

.49 

High  Reading 

190 

.51 

186 

.48 

194 

.64 

190 

.41 

^All  correlations  are  significantly  different  from  zero  (p<C.05). 

^Ns  for  subsamples  will  occasionally  not  add  to  total  for  entire  sample. 
Where  pertinent  data  were  missing  in  Army  records,  a  man  was  omitted  from  an 
analysis. 

^Correlations  with  the  effects  of  MOJ  partialled  out. 


Table  7 

Correlations  of  Job  Sample  and  Job  Knowledge 
for  Category  IV  and  Non-Category  IV  Subgroups^ 


Sample 

Armor 

Crewman 

Repairman 

Supply 

Specialist 

Cook 

r 

N 

r 

N 

r 

N 

r 

Zero-Order  Correlations 

Entire  Sample 

368 

.68 

360 

.59 

380 

.72 

366 

.58 

Category  IV 

186 

.65 

178 

.59 

188 

.69 

186 

.54 

Non-Category  IV 

182 

.66 

182 

.54 

192 

.72 

180 

.54 

Partial  Correlations^ 

Entire  Sample 

368 

.49 

360 

.49 

380 

.65 

366 

.50 

Category  IV 

186 

.34 

178 

.47 

188 

.58 

186 

.42 

Non-Category  IV 

182 

.49 

182 

.45 

192 

.64 

180 

.45 

^All  correlations  are  significantly  different  from  zero  (p^.OB). 

^Ns  for  subsamples  will  occasionally  not  add  to  total  for  entire  sample. 
Where  pertinent  data  were  missing  in  Army  records,  a  man  was  omitted  from 
an  analysis. 

^Correlations  with  the  effects  of  MOJ  partialled  out. 
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Table  8 


Correlations  of  Job  Sample  and  Job  Knowledge 
for  Low  Verbal  and  High  Verbal  Aptitude  Subgroups^ 


Sample 

Armor 

Crewman 

Repairman 

Supply 

Specialist 

Cook 

n'’ 

r 

N 

r 

N 

r 

N 

r 

Zero-Order  Correlations 

Entire  Sample 

368 

.68 

360 

.59 

380 

.72 

366 

.58 

Low  Verbal 

178 

.66 

174 

.55 

174 

.68 

174 

53 

High  Verbal 

181 

.68 

179 

.55 

179 

.72 

186 

59 

Partial  Correlations^ 

Entire  Sample 

368 

.49 

360 

.49 

380 

.65 

366 

.50 

Low  Verbal 

178 

.40 

174 

.44 

174 

.58 

174 

.41 

High  Verbal 

181 

.53 

179 

.48 

179 

.65 

186 

.54 

®AII  correlations  are  significantly  different  from  zero  (p<C.05). 

^Ns  for  subsamples  will  occasionally  not  add  to  total  for  entire  sample. 
Where  pertinent  data  were  missing  in  Army  records,  a  man  was  omitted  from 
an  analysis. 

^Correlations  with  the  effects  of  MOJ  partialled  out. 


Table  9 

Correlations  of  Job  Sample  and  Job  Knowledge 
for  Low  Education  and  High  Education  Level  Subgroups^ 


Sample 

Armor 

Crewman 

Repairman 

Supply 

Specialist 

Cook 

n'^ 

r 

N 

r 

N 

r 

N 

r 

Zero-Order  Correlations 

Entire  Sample 

368 

.68 

360 

.59 

380 

.72 

366 

58 

Low  Education 

142 

.68 

150 

.63 

99 

.77 

140 

.50 

High  Education 

222 

.67 

209 

.56 

281 

.70 

223 

.62 

Partial  Correlations^ 

Entire  Sample 

368 

.49 

360 

.49 

380 

.65 

366 

.50 

Low  Education 

142 

.40 

150 

.44 

99 

.58 

140 

.41 

High  Education 

222 

.53 

209 

.48 

281 

.65 

223 

.54 

®AII  correlations  are  significantly  different  from  zero  (p*<C.05). 

Ns  for  subsamples  will  occasionally  not  add  to  total  for  entire  sample. 
Where  pertinent  data  were  missing  in  Army  records,  a  man  was  omitted  from  an 
analysis. 

^Correlations  with  the  effects  of  MOJ  partialled  out. 


Table  10 


Correlations  of  Job  Sample  and  Job  Knowledge 
for  Black  and  White  Subgroups^ 


Sample 

Armor 

Crewman 

Repairman 

Supply 

Specialist 

Cook 

r 

N 

r 

N 

r 

N 

r 

Zero-Order  Correlations 

Entire  Sample 

368 

.68 

360 

.59 

380 

.72 

366 

.58 

Black 

70 

.80 

46 

.55 

87 

.75 

70 

.59 

White 

294 

.65 

312 

.59 

287 

.72 

289 

.57 

Partial  Correlations^ 

Entire  Sample 

368 

.49 

360 

.49 

380 

.65 

366 

.50 

Black 

70 

.48 

46 

.30 

87 

.64 

70 

.55 

White 

294 

.46 

312 

.51 

287 

.65 

289 

.49 

®AII  correlations  are  significantly  different  from  zero  (p<.05). 

for  subsamples  will  occasionally  not  add  to  total  for  entire  sample. 
Where  pertinent  data  were  missing  in  Army  records,  a  man  was  omitted  from  an 
analysis. 

^Correlations  with  the  effects  of  MOJ  partialled  out. 


In  each  Army  job  studied,  the  zero-order  correlations  for  the  subgroups  are  similar 
to  each  other.  While  the  partial  correlations  (with  months  of  job  experience  partialled 
out)  occasionally  show  somewhat  greater  separation,  suggesting  some  possible  effect  of 
restriction  in  range  on  job  knowledge  test  scores  in  groups  with  lower  reading  ability, 
only  one  (out  of  20)  of  the  differences  between  pairs  of  correlations  for  the  subsamples 
is  statistically  significant.  This  difference  was  found  in  Armor  Crewman  between  the  low 
and  high  reading  groups. 


CONCLUSION 

On  the  basis  of  the  relatively  low  skill  requirements  identified  in  the  analysis  of  job 
sample  test  steps  and  the  high  degree  of  correlation  between  job  sample  test  scores  and 
job  knowledge  test  scores,  it  appears  reasonable  to  conclude  that  in  the  four  occupational 
specialties  studied,  the  job  knowledge  tests  could  appropriately  be  used  in  lieu  of  the  job 
sample  tests. 

As  indicated  earlier,  job  knowledge  tests  can  be  used  instead  of  job  sample  tests  in 
occupational  specialties  if  skill  components  in  such  jobs  are  minimal. 

In  the  development  of  job  knowledge  tests,  it  is  crucial  that  the  tests  require  only 
knowledge  that  is  needed  in  the  actual  performance  of  the  job.  Guidance  for  the 
preparation  of  written  test  items  should  be  given  careful  and  rigorous  attention  (22). 

The  estimates  of  the  lower  limit  of  reliability  for  the  job  sample  tests  indicate  that 
these  tests  are  adequate  where  group  measurement  is  undertaken;  no  conclusion  regarding 
adequacy  of  reliability  for  individual  measurement  can  be  made.  The  high  reliabilities  of 
the  job  knowledge  tests  allow  their  use  where  group  or  individual  measurement  is  desired. 
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Both  the  job  knowledge  tests  and  the  job  sample  tests  possessed  n^liability  characteristics 
making  them  suitable  for  use  in  the  UTILITY  research.  Since  only  lower  limit  reliability 
estimates  are  available,  it  is  not  possible  to  discuss  comparisons  between  the  job 
knowledge  and  job  sample  reliabilities.* 


*It  is  likely  that  the  reliabilities  reported  for  both  the  job  knowledge  and  job  sample  tests 
represent  underestimates  of  the  degree  to  which  they  are  free  of  errors  of  measurement,  since  tests  were 
heterogeneous.  (See  earlier  discussion  of  the  composition  of  the  job  sample  subtests  and  the  examples  of 
the  job  knowledge  test  items.)  Such  an  underestimate  is  particularly  likely  in  the  case  of  the  Cook  job 
sample  test  where  the  subtests  are  dissimilar,  ranging  across  fueling  and  lighting  a  field  range,  preparing 
food,  and  planning  the  preparation  of  a  meal,  including  assignment  of  kitchen  duties,  timing  the  cooking 
sequence,  and  use  of  left-overs  (using  the  Cook’s  Worksheet). 
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I  CG  1ST  CAV  lAIRMOJU^I  ATTN  AC'’'FS  G3  4»Q  SAN  FRAN  963B3 
1  us  ARMY  TROPIC  test  CTR  PO  ORAWER  94R  ATTN  BThAV  SCIEN  C7 
10  CG  111  CORPS  C  FT  HOOD  ATTN  G3  SFC  FT  HOOD 
3D  CO  1ST  ARMORED  OIV  ATTN  C3  SEC  FT  -tQOO 
3J  ^G  2D  ARMPRtD  31V  ATTN  G3  SEC  FT  hOOD 
25  cn  I3TH  cyPT  'BGDF  ATTN  S3  SFC  HOOD 
1  CG  USAFAC  £  FT  SILL  ATTN  AKPSIGT-TNTN  « 

20  CO  111  rORPS  ARTY  ATTN  FB  SEC  FT  SILL 
15  CO  1ST  AIT  BGOE  ATTN  G3  SEC  pt  BLISS 
1  RSCH  CCNTQACTS  E  GRANTS  BR  ARC 

1  BFCD  ART  '^FC  CHF  DF  WASH  DC 

I  CHF  OP  RtD  OA  ATTN  SCI  INFO  « R  RSCH  SPT  DIV  WASH  DF 
1  CINC  US  ATLANTIC  FLT  C^CF  312A  USN  BASE  NORFOLK 
1  COR  TNG  enr^MANO  US  PACIFIC  FLT  SAN  DIEGO 

3  DIR  PPRP  RES  PIV  BUk  PF  NAV  PERS 

I  TECH  lib  our  TP  SHIPS  CODE  2 1 OL  NAVY  DEPT 
1  CO  FLT  ANTI-AIR  WARFARE  TNG  S An  DIcGO 

1  rn  NUCLfAk  WPNS  TNG  CTR  PACIFIC  US  NAV  AIR  STA  SAN  DIEGO 

1  CO  FLEET  TNG  CTR  US  NAV  STA  p AN  DIEGO 

I  cn  FLT  ANT1-SU3  WARFAPP  SCH  SAN  DIEGO 

1  CHF  OF  'AVL  RSCH  PFRS  £  TN^-  6R  ICDQF  455)  ARL  VA 

I  DIR  US  NAV  RFS  LAB  ATTN  COPP  51  2D 

1  DIR  naval  RSFH  lab  ATTN  LIP  CODE  20?®  WASH  DC 

I  CC  MFD  FLO  ocs  l43  CAM®  LEJEUNF 

I  DIR  AfROSPACF  CREW  FOUIP  LA'  NAV  AIR  ENGNR  CTR  PA 
K  COMDT  marine  rnR'»S  hZ  MARINE  C^PPS  ATTN  CODE  AH-IS 

I  HQ  MARINF  CORBS  AT'^N  AX 

1  OIR  MARINF  CORPS  INST  ATTN  PVAL  UNIT 

1  US  MAPINF  CORPS  HQS  HIST  R?F  LIP  ATTN  MRS  JADOT 

I  CHF  OF  NAV  air  Tffw  T  NF,  nAV  AIR  STA  MEMPHIS 

1  DIR  OPS  EVAL  GR°  0F=  Cf  CHF  OF  naV  OPS  DpO‘*EG 

1  CO  US  C^axT  GUARD  TNG  CTR  GOVERNORS  ISLAND  NY 

I  CO  US  CCAST  GUARD  TNG  C7P  CAPE  “AY  NJ 

1  CO  US  COAST  Guard  TNG  CTP  £  SU®  CTR  ALAmfqa  CALU 

I  CO  US  COART  GUARD  INST  CKLA  FITV  DKLA 

1  CO  US  ftaST  FAJARD  RFS  TNG  FT&  Yno.<TCWN  VA 

I  SUPT  US  COAST  GJARD  AfAD  NEW  LONDON  CONN 

I  TECH  ni®  TECH  DIVIHPD)  AFHRL  LDWRy  afR  FOLD 

I  CHF  Sn  OIV  DOFTF  SCI  ♦  TECH  DCS  R  ♦^D  AIR  FORCE  AFRSTA 

I  AFhrl/TT  AT^’N  FAOT  W  S  'ELLMAN  LCWRY  afb 

1  HQ  SA'-ISn  ISMSIR)  AF  unit  post  .?fC  la  Ar  S  CALIF 

2  AFhRL  (hrt)  wR I GHT-p ATTtPSOM  ACR 

1  AMO  A**R‘J  "BDOKS  TEXAS 

2  CF  HU'YAN  PESOUPCfS  lab  RRmKS  AF  a 

I  COMOT  JSAF  SOFT  -3  v,TH  (TAr>  c^lIN  AF  B 
1  af-rl  (PT)  williams  aFB  ARIZ 
I  PSYCHFBinLOGY  NATL  SCI  FOUND 

1  OIR  NATL  SP.CUR  AGy  ft  Gf H  G  MEADE  ATTN  TDL 

3  CU  ATTN  CRS/ADD  STANDARD  DIST 

1  SYS  EVAL  DTV  R  =  S  OMECTCRATE  O'^O-OCD  PENTAGON 
I  0'=PT  CF  STATE  BUR  OP  INTEL  ♦  RES  EXTERNAL  RES  STAFP 

1  SCI  INFO  EYCH  WASHINGTON 

2  CHC  mGT  t  GEN  TNG  OIV  TR  20D  FAA  WASH  OC 

1  8UR  OP  RES  G  ENGR  US  P'^ST  OFC  D*=pT  ATTN  ChP  HUMAN  FACTORS  BR 
I  EDUC  m=d1A  RR  06  HEW  XTTN  T  C  ClEMENS 


1  NAT*L  '^U'^  STANDS  BEHAV  SCI  DP  A^TN  0^^  0  ?  ERLIC-K 

1  OFC  OP  INTPPNATL  TN“,  PLANNING  &  SVAL  AIO  WASH  OC 

1  06PT  TF  TRANS  FAA  AGO  SPG  HQ  ^lOA  WASH  OC 

2  ERIG  PE  WASH  PG 

2  OUNLAP  ♦  ASST'’  I NG  DARIEN  ATTN  LI“ 

2  RAG  ATTN  LIR  '^CLCAN  VA 

1  NITRE  GPRR  BEOFOaO  NASS  ATTN  LIB 

2  LEARNING  R&O  GTR  U  OF  PITTS  ATTN  DIR 

2  TEGH  INFO  GTR  PNSNR  OATA  SERV  N  ANER  AVN  ING  GOLUNRUS  P 

I  GHRYSLEP  CORP  MSL  OIV  OFTRPIT  ATTN  TECH  INFO  GTR 

I  GEN  OVNA’-IGS  POMONA  OIV  ATTN  LIP  OIV  CALIF 

I  MGR  BIOTECHNDL^SV  AEROSPACE  SVS  OIV  MS  BH-25  ROriNO  CO  SEATTLE 
I  lOA  RSCH  G  ENG  SUPT  OIV  ARL  VA 

1  SCI  £  TPCH  OIV  lOA  ARL  VA 

I  HUGHES  AIRCRAFT  COMPANY  CULVER  CITY  CALIF 
1  OIR  GTR  FOR  RES  ON  LEARNING  ♦  TEACHING  U  3"  MICH 

1  EOITQo  TNG  RFS  ABSTR  AMER  SOC  OF  TNG  DIPS  U  OF  TENN 

3  AUSTRALIAN  NAV  ATTACHE  EMBSY  OF  AUSTRALIA  WASH  OC 

2  AUSTRALIAN  ARMY  ATTACHE  EMpSY  OF  AUSTRALIA  ATTN  TECH  CL< 

I  MENNINGER  FOUNDATION  TOPEKA 

I  AMCR  InSTS  for  RSCH  SILVER  SPRING 
I  AMER  INSTS  FOR  RSCH  ATTN  LIB\  PA 
1  OIR  PPiMATc  LAB  UNIV  PF  WiS  MADISON 
I  EOUC  £  TNG  f.ONSLT  CO  LA  CALIF 

I  OR  GEORGE  T  HAUTY  CHMN  pFPT  OF  PSYCHOL  U  OF  OEL 
1  HEAO  OEPT  OF  PSYCHOL  UNIV  OF  SC  COLUMPIA 
I  U  OF  GEORGIA  OEPT  OF  PSYCHOL 
I  G£  CO  WASH  0  C 

I  AMER  INST  FOP  RSCH  ATTN  LIP  PALO  ALTO  CALIF 
I  ROWLAND  ♦  CO  HADOONFIELT  NJ  ATTN  oRES 

1  OHIO  STATE  U  SCH  OF  AVN 

2  aircraft  ARMAMENTS  INC  COCKEYSVILLE  Ml 

I  OR  J  B  CULLEN  DEPT  OF  SCC  £  ANTHROP  UNIV  OF  RI 
1  AMER  PSYCHOL  ASS'LL  WASHINGTON  ATTN  PSYCHOL  ABSTR 
I  NO  ILL  U  HEAD  DEPT  OF  PSY'-HOL 
1  GEORGIA  INST  n*'  IFfH  OIR  SCH  OF  PSYCHOL 
I  LIFE  SCI  INC  H'JRST  TEXAS  ATTN  W  G  MATHENY 
I  AMER  REHAV  SCI  calif 

1  SO  ILLINOIS  U  D=PT  CF  PSYCHOL 

2  WASH  military  SYS  TECH  LIR  DIV  PETHESDA  MD 
1  NORTHWESTERN  U  DEPT  OF  INOSTR  PNONR 

I  OR  L  TWYFPRO  NY  STATE  EOUC  OEPT  ABSTRACT  EOITOP  AVCR 
I  AEROSPACE  SAFETY  DIV  U  CF  SOUTHERN  CALIF  LA 
I  MR  BRANOON  B  SMITH  R^S  ASSOC  U  OF  MINN 

1  OR  V  2ACHERT  RT  I  GOOD  HOPE  GA 

I  OR  S  ROSCOE  ASSOC  DIR  pCR  RSCH  INST  OF  AVN  U  OF  ILL 

I  OR  C  HELM  DEBT  EOUC  PS>^CH  CITY  U  OF  NY 

I  GEN  H  p  MAURIS  lUSA  RETIPRES  THE  CITAOEL  SC 
1  OR  H  SHPt»'AKfR  DIR  TNG  RSCH  GP  NY 

I  VOC-TtCH  EOUC  PROG  PLNNO  DEV  ATTN  W  STOCK  ST  PAUL 
I  CHF  PROCESSING  OIV  DUKE  U  LIB 
1  U  OF  CALIF  OtN  LIB  COCU  DEPT 
1  PSYCHOL  LIR  HARVARD  UNIV  CAMBRIDGE 

1  U  OF  ILL  LIR  SER  OEPT 

2  U  OF  KANSAS  LIB  pEPIOOICAL  OEPT 
I  U  OF  NEBRASKA  LIPS  ACO  Of^T 


I  OHIO  STATE  U  LIBS  GIFT  ♦  ^XCH  DIV 
I  PFNNA  STATE  U  PATTEE  LIP  DOCU  DESK 
1  PURDUE  U  LIBS  PERIOCICAIS  CHECKING  FILES 
I  STANFORD  U  LIPS  X3CU  LIP 
1  LIPN  U  re  TFXAS 
I  SYRACUSE  U  LIR  SER  OIV 
I  SFRIALS  REC  UNIV  OF  MINN  MINNEAPOLIS 
I  STATE  U  OF  IOWA  LIBS  SER  ACQ 

1  NO  CAROLINA  STATE  COLL  OH  HILL  LIB 

2  BOSTON  U  LIRS  ACQ  OIV 

1  U  0=  mKh  lips  SER  DIV 
1  PR  OWN  U  LIR 
I  COLUMBIA  U  LIBS  DOCU  ACO 

1  DIR  JOINT  U  LIBS  NASHVILLE 

2  LP  GEO  WASH  UNIV  ATTN  SPEC  CDlL  DEPT  WASH  OC 
2  LIR  OF  CONGRESS  CHF  OF  EXCH  ♦  GIFT  DlV 

1  U  OF  PGH  DOCU  LIRN 

I  CATHOLIC  U  LIR  EDIJC  £  PSYCH'LL  LIB  WASH  OC. 

1  U  QB  KY  MARGARET  I  KING  LP 
I  SO  ILL  U  ATTN  LPN  SCR  DEPT 
I  BRIGHAM  YOUNG  U  LIR  SFR  SECT 
I  U  OF  LOUISVILLE  LIR  BELKNAP  CA-PUS 
1  GEORGETOWN  U  LP  SER  OEPT  WASH  OC 
I  LIBS  COLO  STATE  U  ATTN  OOC  LIRN  FT  ^nLLINS 

I  CLIN  PSYr.HDL  SEPV  OEPT  OF  NEURF'PRVCH  UT  WALTER  REEO  G^n  HOSP 
I  CO  OEF  OEV  ENGR  LAP  ATTN  LPN  EOGEWOOD  ARSNL  MO 

1  CO  USA  COC  CPP  ACWY  FT  MCCLELLAN  ALA 

I  CG  army  ELEC  PVG  CO  ATTN  TECH  LIB  FTHUACHJCA  ARP 

I  CHF  OFPT  OF  CLIN  £  SOC  PSYCH  W^AIR  WASH  OC 
I  CO  OUGWAY  pvG  go  ATTN  TECH  LIR  U^AH 

I  CO  USA  mobility  fquIP  R£0  CTB  ATTN  TECH  OOC  CTR  FT  “j-lvOIR 
I  CG  2N0  RGM  ARAOCOM  R IC HARDS-GER AUP  AF R  MD 
1  PEPS  SURSYS  OIV  CREW  SUFSYS  ORCT  AEKC  SvS  OIV  WPIC.HT-oat. 

I  CO  USACOC  fBT  SUPPORT  GP  FT  PELVOIR 
1  CG  CPNARC  ATTN  DCS  INTEll  SCH  FT  HUACHUCA 
I  COMDT  INOSTR  COLL  OF  THE  ARMFQ  FORCES  FT  MCNAIR 

I  COMOT  NATL  WAR  COLL  ft  LESLEY  J  MCNAP  ATTN  CLASSE  RECORDS  “R  LIB 

1  COMOT  USA  MGT  SCH  FT  BELVCIR 

1  LEADERSHIP  CCM  CO  0®$  OEPT  USAIS  FT  PENNING 

1  OIR  COMPANY  TACTICS  DFPT  USAIS  FT  RSNNING 

I  CHF  OF  NAV  PSCH  ATTN  DIR  PSYCHOL  SCI  OIV  COOE  ASO 

I  CHF  CURRICLUM  Ba  RESrOENT  INSTR  DEPT  USA  LOG  “Gmy  CT  FT  LEE 

I  LIB  US  NAV  MED  PSCH  LAB  USN  SUBMARINE  RASE  GROTON 

1  RAOC  RASH  GRIpeiSS  APR  NY 

I  ATC  ATXRO  RANOOLPH  AFP 

1  OTR  RAND  CORP  ATTN  LIP  WASH  OC 

I  CRESS-AIR  ATTN  LIRN  **0 

1  BATTELF  MEMORIAL  INST  ATTN  RACIC  COLUMBUS  GA 
1  CO  USN  SCH  OF  AVN  mfq  ySN  AVN  MFO  CTR  FLA 
I  CO  £  OIR  USN  ELECT  LAP  ATTN  LIP  SAN  OIEGO 
I  SMAMA  (SMACU-PRSNL  RSCH)  MCCLELLAN  AF R  CALIF 
1  CO'^0  ELECT  SYS  DIV  ATTN  ESTI  Rfopord  MASS 

I  GP  EFFFCTIVENESS  RSCH  LAR  U  OF  ILL  DEPT  OF  osyCHCL 
I  AV^n  COPP  Avro  WSL  OIV  ATTN  RSFh  LP  DETROIT 
1  1  EACH  USA  TNG  CTR 

I  US  OEPT  DE  LAROR 
I  US  GOVT  »'FC  WASH  OC 
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